Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 407513 reports (Page 5166 of 8151)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X.....XX......X...........................................] duration: 2.450888624s FAIL (2025-09-03 21:20:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.450921963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X.....XX......X..........................................] duration: 2.890817489s FAIL (2025-09-03 21:20:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.890871059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.984578032s FAIL (2025-09-03 21:20:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.984642895s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.894862372s FAIL (2025-09-03 21:19:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.894888688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.....XX......X.........................................] duration: 2.490928642s FAIL (2025-09-03 21:19:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.490969964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 2.905931021s FAIL (2025-09-03 21:19:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.905957492s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.747961108s FAIL (2025-09-03 21:18:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.74801477s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X.....XX......X........................................] duration: 3.335019722s FAIL (2025-09-03 21:18:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.335065786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 2.966448794s FAIL (2025-09-03 21:18:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.966477607s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.752456017s FAIL (2025-09-03 21:18:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752480088s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X.....XX......X.......................................] duration: 2.915149428s FAIL (2025-09-03 21:17:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.915188089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 2.987965466s FAIL (2025-09-03 21:17:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.987988415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X.....XX......X......................................] duration: 3.906171739s FAIL (2025-09-03 21:17:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.906205824s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 2.869466622s FAIL (2025-09-03 21:17:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.869509792s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 3.159396106s FAIL (2025-09-03 21:16:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.159418565s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X.....XX......X.....................................] duration: 2.842661619s FAIL (2025-09-03 21:16:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.842708002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 3.057352264s FAIL (2025-09-03 21:15:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.058306968s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X.....XX......X....................................] duration: 2.778977067s FAIL (2025-09-03 21:15:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.779015621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.934112709s FAIL (2025-09-03 21:14:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.934136884s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.X.....XX......X...................................] duration: 2.859054321s FAIL (2025-09-03 21:14:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.859100442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.848109487s FAIL (2025-09-03 21:14:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.848211553s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.X.....XX......X..................................] duration: 2.373088366s FAIL (2025-09-03 21:13:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.373131224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 2.821631627s FAIL (2025-09-03 21:13:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.821781202s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 2.972934774s FAIL (2025-09-03 21:13:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.972956175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.X.....XX......X.................................] duration: 2.684695276s FAIL (2025-09-03 21:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.684729238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 2.774590414s FAIL (2025-09-03 21:12:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.774625423s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 2.824174954s FAIL (2025-09-03 21:11:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.824550424s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.X.....XX......X................................] duration: 2.679148675s FAIL (2025-09-03 21:11:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.679182674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.X.....XX......X...............................] duration: 2.471674212s FAIL (2025-09-03 21:10:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.471774811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...........................XX..............................] duration: 3.471165862s FAIL (2025-09-03 21:10:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.471194416s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.X.....XX......X..............................] duration: 3.041338664s FAIL (2025-09-03 21:09:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.041359807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.X.....XX......X.............................] duration: 2.733357206s FAIL (2025-09-03 21:09:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.733387895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX...........................XX.............................] duration: 3.113898976s FAIL (2025-09-03 21:09:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.113939394s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.X.....XX......X............................] duration: 2.551883743s FAIL (2025-09-03 21:09:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.551944033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX...........................XX............................] duration: 2.782784891s FAIL (2025-09-03 21:08:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.782816492s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX...........................XX...........................] duration: 3.224379668s FAIL (2025-09-03 21:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.224413045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.X.....XX......X...........................] duration: 3.282619954s FAIL (2025-09-03 21:08:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.282652054s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX...........................XX..........................] duration: 2.982433803s FAIL (2025-09-03 21:07:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.983059592s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.X.....XX......X..........................] duration: 3.044689222s FAIL (2025-09-03 21:07:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.044721854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX...........................XX.........................] duration: 2.709938006s FAIL (2025-09-03 21:07:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.710036115s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX...........................XX........................] duration: 4.100628892s FAIL (2025-09-03 21:06:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.100659389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.X.....XX......X.........................] duration: 3.578515155s FAIL (2025-09-03 21:06:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.578555572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..................X.X.....XX......X........................] duration: 3.508690516s FAIL (2025-09-03 21:05:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.508723997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX...........................XX.......................] duration: 3.29423588s FAIL (2025-09-03 21:05:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.294518799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..................X.X.....XX......X.......................] duration: 2.690047685s FAIL (2025-09-03 21:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.690079924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.658587038s FAIL (2025-09-03 21:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.658622359s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX...........................XX......................] duration: 3.039378546s FAIL (2025-09-03 21:04:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.039403123s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..................X.X.....XX......X......................] duration: 3.220345178s FAIL (2025-09-03 21:03:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.220375947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.931728973s FAIL (2025-09-03 21:03:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.931758779s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX...........................XX.....................] duration: 3.305510788s FAIL (2025-09-03 21:02:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.305537131s edge gateway us-select-1

1
...
5164
5165
5166
5167
5168
...
8151