Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X..................X.X.....XX......X....] duration: 2.681671493s FAIL (2025-09-03 20:51:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.681708882s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....XXX.XXX.XX.X........XX...........................XX....] duration: 2.864080586s FAIL (2025-09-03 20:50:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.864108058s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......XXX.XXX.XX.X........XX...........................XX...] duration: 2.838436111s FAIL (2025-09-03 20:50:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.838466858s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X......X....X........................................] duration: 2.576651242s FAIL (2025-09-03 20:50:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.576752971s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......XXX.XXX.XX.X........XX...........................XX..] duration: 2.918239218s FAIL (2025-09-03 20:50:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.918271235s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X..................X.X.....XX......X...] duration: 2.591112756s FAIL (2025-09-03 20:49:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.591154362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX......X......X....X.......................................] duration: 2.512772472s FAIL (2025-09-03 20:49:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.512803829s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XXX.XXX.XX.X........XX...........................XX.] duration: 3.051295867s FAIL (2025-09-03 20:49:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.051358876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX......X......X....X......................................] duration: 2.472239697s FAIL (2025-09-03 20:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.472343895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X..................X.X.....XX......X..] duration: 3.332630881s FAIL (2025-09-03 20:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.332669021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........XXX.XXX.XX.X........XX...........................XX] duration: 30.001396273s FAIL (2025-09-03 20:48:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001423768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X..................X.X.....XX......X.] duration: 3.479679959s FAIL (2025-09-03 20:47:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.47971906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X..................X.X.....XX......X] duration: 1m0.103108942s FAIL (2025-09-03 20:47:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3 20:46:51.711096921 (126h39m12.203529366s) [upload=111516]: Got next segment. Inline: false Sep 3 20:46:51.711135876 (126h39m12.203568309s) [upload=111516]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:46:51.711141270 (126h39m12.203573705s) [upload=111516]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:46:51.906736001 (126h39m12.39916845s) [seg_pos={0 0}, upload=111516]: Begin upload segment... Sep 3 20:46:51.906793468 (126h39m12.39922591s) [seg_pos={0 0}, upload=111516]: Done begin upload segment. Sep 3 20:46:51.906846069 (126h39m12.399278514s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907276279 (126h39m12.399708726s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907648609 (126h39m12.400081055s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.907964168 (126h39m12.40039661s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908406706 (126h39m12.400839153s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908699347 (126h39m12.401131794s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.908971584 (126h39m12.401404025s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909264539 (126h39m12.401696983s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909559058 (126h39m12.401991503s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.909956083 (126h39m12.402388525s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910336771 (126h39m12.402769216s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910602378 (126h39m12.403034824s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.910832624 (126h39m12.403265074s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911075069 (126h39m12.403507515s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911299238 (126h39m12.403731689s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911579424 (126h39m12.404011874s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.911950332 (126h39m12.404382794s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:51.912237363 (126h39m12.404669811s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Uploading piece... Sep 3 20:46:52.481805815 (126h39m12.974238262s) [upload=111516]: Next returned nil segment Sep 3 20:46:52.481831685 (126h39m12.974264146s) [upload=111516]: Waiting for error group managing segments... Sep 3 20:46:52.639635577 (126h39m13.132068011s) [noise=true , address=45.91.116.8:4049 , piece_id=36TQFYOBSNNTARLHHVZYQV72EOP4KT7GT227WGZ6HSD62WSQDSRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.649144857 (126h39m13.141577285s) [noise=true , address=77.90.31.16:4747 , piece_id=GJDR3BCLKSTIAEQM3BEBDPS3LTNZDAUAJRTFYNHYAL32CRVBHONQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.696435343 (126h39m13.188867791s) [noise=true , address=77.90.31.13:4625 , piece_id=JE2VPVFJ45INP3N6ROE2IKWKFH5W3ZRADELV5GQZS3O2I5OSGSOA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.699823249 (126h39m13.192255685s) [noise=true , address=104.36.82.60:4342 , piece_id=J3K5EIDU6ATX4SNNVA3BW63DDE5ZRGCAUMGFXN72LSH63XMXRKAQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.728980202 (126h39m13.221412649s) [noise=true , address=45.91.116.11:4121 , piece_id=DAZCJ3BGGDUKQCRWGZDQMKP7AKBXLVJRYXFMCMS423OMAW7MQHRA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.736349395 (126h39m13.22878183s) [noise=true , address=77.90.31.19:4872 , piece_id=CXRHCKIGZJTZ43UNQ7CKTUOWRJSR5EJCYVFTUGU7VTHBLFBFRMLA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.772619194 (126h39m13.265051623s) [noise=true , address=77.90.31.15:4715 , piece_id=ON3XS5AIWNEXCG2ADRRYI47E5KBO7W4EQ2P43VDPVPQJBS25N6XA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:52.982474880 (126h39m13.474907309s) [noise=true , address=104.36.82.23:4446 , piece_id=YI6B5R5KIGCMUS5IKZN5XXGN4CIODHJEZSIMOIPFZWDRT62QJY3Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.036894411 (126h39m13.529326859s) [noise=true , address=104.36.82.59:4326 , piece_id=FNNQWNSI6RLKIMFPNWVCJHTVXS4LE7ZECRVE5RLDRM4OINLNUBMQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.185205565 (126h39m13.677637993s) [noise=true , address=64.49.13.137:4498 , piece_id=TLZMVZYAMMNVZOQ7YVW5FLI7YSFCRAWURNGODA4VMGKBGV73OV7A, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.317694553 (126h39m13.810126987s) [noise=true , address=79.127.205.245:4233 , piece_id=XATH7HNND2GRWF4WJDAMXGQCOYFY7E6JIJ6GKPOXHIWICH6J6P6Q, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:46:53.821769162 (126h39m14.314201595s) [noise=true , address=77.90.31.14:4689 , piece_id=QPTIL7MCSO5XVS43LZQQBCGSZKZSDEMNUGZFTIPAZODUPPUZ7UAA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:26.655268500 (126h39m47.147700932s) [noise=true , address=77.90.31.10:4550 , piece_id=L4PF7BPY4CX77Z6UPO3FTUTAZB7CRQEAPQA2QNWXHH3OXR567PYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: <nil> Sep 3 20:47:51.443014263 (126h40m11.935446706s) [noise=true , address=104.36.82.55:4210 , piece_id=4SF533GBR3GKIDPCS4FIKZ4GLCW5RQ5GTX6QNKQXW2PLU27XCPFA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 139.59.144.22:51206->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443097716 (126h40m11.93553015s) [noise=true , address=104.36.82.43:4843 , piece_id=WZ4R6ZU3AAVGYXIW6QX3A55YMLFPMCDZ25UIEXOTQREBN6YMFXRQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:1bBR6KYQ1sSfvunbVsY8t12xrN6uk2kjq8ccDcFiopYSX72tet, address:104.36.82.43:4843): protocol: rpc: dial tcp 104.36.82.43:4843: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443199202 (126h40m11.935631635s) [noise=true , address=104.36.82.34:4570 , piece_id=STMTXDPIMVVFWBRSZ7XXALPB3ESD342RWZ7BDMTTTJWPBK5H4ZZA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443267164 (126h40m11.935699598s) [noise=true , address=104.36.82.55:4215 , piece_id=LBPB53PVKOG45D2TDZ4TCCOG456RUZENZFUQJFHY4633JHZSGJYQ, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:47:51.443413567 (126h40m11.935845995s) [noise=true , address=104.36.82.47:4532 , piece_id=5GBMJXQP3NEHLDPZAINM3E5BS5CF3TWSSQ7W7ZMD5S6JEABXCIEA, seg_pos={0 0}, upload=111516]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 3 20:47:51.443461047 (126h40m11.935893496s) [seg_pos={0 0}, upload=111516]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:47:51.544201118 (126h40m12.036633584s) [upload=111516]: Done uploading segments... Sep 3 20:47:51.544237317 (126h40m12.036669766s) [upload=111516]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.103202587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX......X......X....X.....................................] duration: 2.5296258s FAIL (2025-09-03 20:47:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.529657618s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.........XXX.XXX.XX.X........XX...........................X] duration: 30.001451654s FAIL (2025-09-03 20:47:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001480619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX......X......X....X....................................] duration: 2.387777513s FAIL (2025-09-03 20:47:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38781502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX......X......X....X...................................] duration: 2.520037143s FAIL (2025-09-03 20:46:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.520067091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..................X.X.....XX......] duration: 2.812754216s FAIL (2025-09-03 20:46:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.812784263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.........XXX.XXX.XX.X........XX...........................] duration: 3.233194573s FAIL (2025-09-03 20:46:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.233226374s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..................X.X.....XX.....] duration: 3.852967557s FAIL (2025-09-03 20:45:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.853019429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX......X......X....X..................................] duration: 2.36766036s FAIL (2025-09-03 20:45:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.367699839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..................X.X.....XX....] duration: 4.045142536s FAIL (2025-09-03 20:45:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 4.045175068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXX.........XXX.XXX.XX.X........XX..........................] duration: 2.914699114s FAIL (2025-09-03 20:45:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.914730198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX......X......X....X.................................] duration: 2.370449108s FAIL (2025-09-03 20:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370554614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X..................X.X.....XX...] duration: 2.932613979s FAIL (2025-09-03 20:44:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.932653852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX.........XXX.XXX.XX.X........XX.........................] duration: 2.750003092s FAIL (2025-09-03 20:44:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.75004329s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXX.........XXX.XXX.XX.X........XX........................] duration: 2.749712832s FAIL (2025-09-03 20:43:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.749732478s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X..................X.X.....XX..] duration: 3.652066114s FAIL (2025-09-03 20:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.652100515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X......XX......X......X....X................................] duration: 3.607095929s FAIL (2025-09-03 20:43:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.607127015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X..................X.X.....XX.] duration: 3.251506843s FAIL (2025-09-03 20:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.251541055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X..................X.X.....XX] duration: 1m0.140944717s FAIL (2025-09-03 20:42:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 3 20:41:57.964175106 (126h34m18.45660755s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.964586684 (126h34m18.457019133s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965081071 (126h34m18.457513513s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965369596 (126h34m18.457802049s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.965741617 (126h34m18.458174062s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966074154 (126h34m18.458506597s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966413444 (126h34m18.458845873s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966683817 (126h34m18.459116245s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.966950819 (126h34m18.45938326s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967210186 (126h34m18.459642627s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967480374 (126h34m18.459912803s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.967739590 (126h34m18.460172038s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970600282 (126h34m18.463032733s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.970988186 (126h34m18.463420638s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971238367 (126h34m18.46367081s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971483038 (126h34m18.463915481s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971723802 (126h34m18.464156252s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:57.971953237 (126h34m18.464385683s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Uploading piece... Sep 3 20:41:58.528419564 (126h34m19.020851999s) [upload=111441]: Next returned nil segment Sep 3 20:41:58.528473137 (126h34m19.02090559s) [upload=111441]: Waiting for error group managing segments... Sep 3 20:41:58.703119733 (126h34m19.19555217s) [noise=true , address=104.36.82.59:4303 , piece_id=NOBZEQSFUXEO22ZOORWBJRSS7CDJNKJ2YTNZ7662BALBWDPDM6WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.714248407 (126h34m19.206680842s) [noise=true , address=104.36.82.23:4465 , piece_id=D37KHQX7CH3LK6BVDCJGJVQ2OL62CWMRRACICJ3AG4EXWXJ2BD3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.738400926 (126h34m19.230833367s) [noise=true , address=45.91.116.6:4990 , piece_id=MXRXONSUW7MVL5GHOPEPH7ZSRBB74FHYDCTVIPDNKBT3FVQJT5KQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.749035563 (126h34m19.241467998s) [noise=true , address=104.36.82.54:4167 , piece_id=QQDOMTBRBVWSQNZOONXMI4VJVMMDF5ERWJF3P25MMEGA32NRI45A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.757249706 (126h34m19.249682152s) [noise=true , address=77.90.31.18:4821 , piece_id=IPNCE5KCEGJLBXWEBQBM7JPCDG2EJVBVXUCSWV4DIURPZLZW5FVQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.770824770 (126h34m19.263257213s) [noise=true , address=104.36.82.62:4442 , piece_id=SB46IH4DFYS5TBKHFTOSO5KP6ILLGIZ7JNDRYJ4XK5BFJ4IK22WQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.839539325 (126h34m19.331971774s) [noise=true , address=104.36.82.45:4925 , piece_id=TYZUAULKHNMLJGU37IYPXEKGAUZGUVPJWUCCI7OHBE7AFMLTMAWA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.853765304 (126h34m19.34619774s) [noise=true , address=104.36.82.12:4177 , piece_id=JS5BA2BZJZVKLKC6JL4DUUE6XONR5INBEJ6MSA3LWA3YRKASIH3Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:58.854122772 (126h34m19.3465552s) [noise=true , address=45.91.116.9:4278 , piece_id=OSRHSKVVFFO5VKQVYEEOS46Q44YVRB6DZJK3IMTRLWRRSOE3FNPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.248391781 (126h34m19.740824215s) [noise=true , address=64.49.13.137:4498 , piece_id=A2QGZOVPLSFZHNM2CEZOIL3CQNA2G2FXCEINHUCN3T6BZ24XCJPQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:41:59.763786820 (126h34m20.256219254s) [noise=true , address=104.36.82.60:4363 , piece_id=QW2COY5Z2Z24HTBHS62OVVJJCTPUR2CHCZG2GCWUKP22BFO2FBLA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: <nil> Sep 3 20:42:57.476354403 (126h35m17.968786833s) [noise=true , address=104.36.82.53:4134 , piece_id=RYHZVCINWDEENIP5SVLR5XSKZZLTMAE2SQTXE3ZYEULK53VEVA5A, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 139.59.144.22:58650->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476411363 (126h35m17.968843792s) [noise=true , address=77.90.31.19:4873 , piece_id=AP46AOFYKE4LTSHSSN62WV52EXWIHBAOZD6CYSSVBHHZO7DNORHQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:122JpURwWNSDakKSHma7tqooTbiTewq6RQcCD8avQX8SKGeRcYV, address:77.90.31.19:4873): protocol: write tcp 139.59.144.22:45360->77.90.31.19:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476459013 (126h35m17.968891442s) [noise=true , address=104.36.82.11:4131 , piece_id=TGCN6NHTMXEXUQSRY34GMX2EP6AQ2HFMF2LS32N44EDWFOZSNNOQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 139.59.144.22:48774->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476481551 (126h35m17.968913996s) [noise=true , address=104.36.82.15:4282 , piece_id=3PYUF454FULYKCFJ72O6CEMIGFO23KXBGQSJJCGFOTNW7TXPA6FQ, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: rpc: dial tcp 104.36.82.15:4282: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476499676 (126h35m17.968932104s) [noise=true , address=104.36.82.55:4228 , piece_id=GVDWQQOI2JLF6GU4DDCK4WSTHABQZJUZKWBG6BOAMKRJY6QMQHFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 139.59.144.22:49698->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476536493 (126h35m17.968968921s) [noise=true , address=104.36.82.41:4768 , piece_id=JOA4HQVPF7YD2EHKVCYCONTXOC3CDPHCTUZ6BX4MMQXLCK7DGYFA, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476578506 (126h35m17.969010941s) [noise=true , address=104.36.82.52:4115 , piece_id=H6BEZWUUSCSYG2FFKADHWHXZHGMIV7AR5ASFFP6Y6KMJN3SL6W4Q, seg_pos={0 0}, upload=111441]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:42:57.476606353 (126h35m17.969038795s) [seg_pos={0 0}, upload=111441]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 3 20:42:57.615750543 (126h35m18.108182975s) [upload=111441]: Done uploading segments... Sep 3 20:42:57.615769896 (126h35m18.108202324s) [upload=111441]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.140988332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XXX.........XXX.XXX.XX.X........XX.......................] duration: 2.681796107s FAIL (2025-09-03 20:42:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.681832652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X......XX......X......X....X...............................] duration: 2.701296862s FAIL (2025-09-03 20:42:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.701332572s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X.XXX.........XXX.XXX.XX.X........XX......................] duration: 2.930579242s FAIL (2025-09-03 20:42:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.93067443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X..................X.X.....X] duration: 1m0.107764865s FAIL (2025-09-03 20:41:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 785271 (126h33m5.28321772s) [upload=111421]: Got next segment. Inline: false Sep 3 20:40:44.790865291 (126h33m5.283297733s) [upload=111421]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:40:44.790873092 (126h33m5.283305544s) [upload=111421]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:40:44.987476967 (126h33m5.479909396s) [seg_pos={0 0}, upload=111421]: Begin upload segment... Sep 3 20:40:44.987541204 (126h33m5.479973632s) [seg_pos={0 0}, upload=111421]: Done begin upload segment. Sep 3 20:40:44.987570461 (126h33m5.480002902s) [noise=true , address=45.91.116.12:4239 , piece_id=2ZKX6L4PTLI3KHEJ57UTJSBSOIUQ7VO7FERARWZ4WWQAUVCEJCKA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.987594798 (126h33m5.480027251s) [noise=true , address=104.36.82.54:4161 , piece_id=DEMOPQFFJSAVATDV6OBCNOP5P2LXSXT3QGMZANIVGVPNTFCHNVUA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.987986295 (126h33m5.480418737s) [noise=true , address=104.36.82.59:4315 , piece_id=BT6AWQL6E4VICOMTLCBJA3VXWVKKCGB2FHRVTX36HZE7ODBJ5UNA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.987999531 (126h33m5.480431978s) [noise=true , address=77.90.31.14:4693 , piece_id=7EYSP5JVSQDRDW2RF77ILMFXHWSOR4AM6UPBEWGWVWBGRSHQGGHA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.988382984 (126h33m5.480815427s) [noise=true , address=104.36.82.62:4439 , piece_id=POLES4KHDMZ7NTAT6UCQSL4HSIDEZQWRWW5DUVESRR2LT3RS4NIA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.988427185 (126h33m5.480859635s) [noise=true , address=77.90.31.21:4906 , piece_id=MNWCM2EF57RYNAWCCF2OUCNXPKBFWSZORP2TLUQKPP6QIPDBH3JA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.988745912 (126h33m5.481178354s) [noise=true , address=104.36.82.60:4353 , piece_id=SDIDG4BUHSRRCXYFNAYHHSO7GKUJIKQMHS443QHEJ4UY25XZ7NMA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.988764417 (126h33m5.481196864s) [noise=true , address=64.49.13.170:4720 , piece_id=6YHASHXKEEXLT2GPT6MV6WMBYWRWXMMQOQHU2OL3GFJNEMN6MYXQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989064896 (126h33m5.481497348s) [noise=true , address=104.36.82.3:4784 , piece_id=Z74VROHRBDITLC7YQFJXZIZ3IX5RHX3S7RQJJHCYSXK5AOLRZYJQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989080478 (126h33m5.48151292s) [noise=true , address=64.49.13.155:4919 , piece_id=3PMM67J3L3FQ5JC6FAXZ7QYGWK4QPQ74YOYCO6LDMMR2TG2WDKTQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989356336 (126h33m5.481788781s) [noise=true , address=77.90.31.15:4728 , piece_id=6KL7X5YMVKQ5AFY4VAEEPMD6EFMUZRKZURWU5UUOI5XVFHJODQSA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989443586 (126h33m5.481876028s) [noise=true , address=45.91.116.8:4051 , piece_id=EG7KBQCCQNUDX4WAZN6Z3MV3P42L3YUKJ7L5YVDMIKCS4S2XCZSA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989641440 (126h33m5.482073885s) [noise=true , address=104.36.82.55:4225 , piece_id=P2HBYO2EI7WCY4WFVWBZLESCIQKUEPSF23LANIHJVXYI7WMJXGXA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.989784820 (126h33m5.482217263s) [noise=true , address=45.91.116.7:4015 , piece_id=TZQWM4N5URLIHQOO6D7X53GKDRA3GWDY3GTB2OPUENPTJWR25XUA, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.990193095 (126h33m5.48262554s) [noise=true , address=79.127.205.246:4279 , piece_id=2T5XZROXE3NL65TT52NYPR5NLHO4446B3SFGCPLGIF52MZ35W27A, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.990555084 (126h33m5.482987526s) [noise=true , address=104.36.82.55:4213 , piece_id=XBGRAYKLN47UGSK3TO4MXWCF7Z4C4YYIZVWBRU5XY7V53TU3FAVQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.991207781 (126h33m5.483640215s) [noise=true , address=77.90.31.19:4862 , piece_id=SJEOAHGJUBIKZHCRN5RYKHFP5SGQGLC7GWTOAQLTMLDNKJYFLSFQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:44.992044246 (126h33m5.484476679s) [noise=true , address=104.36.82.51:4062 , piece_id=2ZRRVJOOXFCIDMMST7FW6JPFXITXV26YSTR56CIVRPTFNEN6W5RQ, seg_pos={0 0}, upload=111421]: Uploading piece... Sep 3 20:40:45.549529464 (126h33m6.0419619s) [upload=111421]: Next returned nil segment Sep 3 20:40:45.549547885 (126h33m6.041980327s) [upload=111421]: Waiting for error group managing segments... Sep 3 20:40:45.729367091 (126h33m6.221799522s) [noise=true , address=104.36.82.54:4161 , piece_id=DEMOPQFFJSAVATDV6OBCNOP5P2LXSXT3QGMZANIVGVPNTFCHNVUA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.745337422 (126h33m6.237769856s) [noise=true , address=45.91.116.7:4015 , piece_id=TZQWM4N5URLIHQOO6D7X53GKDRA3GWDY3GTB2OPUENPTJWR25XUA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.752388782 (126h33m6.244821213s) [noise=true , address=45.91.116.12:4239 , piece_id=2ZKX6L4PTLI3KHEJ57UTJSBSOIUQ7VO7FERARWZ4WWQAUVCEJCKA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.769437174 (126h33m6.261869603s) [noise=true , address=104.36.82.60:4353 , piece_id=SDIDG4BUHSRRCXYFNAYHHSO7GKUJIKQMHS443QHEJ4UY25XZ7NMA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.771703633 (126h33m6.264136068s) [noise=true , address=45.91.116.8:4051 , piece_id=EG7KBQCCQNUDX4WAZN6Z3MV3P42L3YUKJ7L5YVDMIKCS4S2XCZSA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.773197307 (126h33m6.265629742s) [noise=true , address=77.90.31.21:4906 , piece_id=MNWCM2EF57RYNAWCCF2OUCNXPKBFWSZORP2TLUQKPP6QIPDBH3JA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.780127532 (126h33m6.272559965s) [noise=true , address=77.90.31.14:4693 , piece_id=7EYSP5JVSQDRDW2RF77ILMFXHWSOR4AM6UPBEWGWVWBGRSHQGGHA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.826108622 (126h33m6.318541059s) [noise=true , address=77.90.31.15:4728 , piece_id=6KL7X5YMVKQ5AFY4VAEEPMD6EFMUZRKZURWU5UUOI5XVFHJODQSA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.850303079 (126h33m6.342735508s) [noise=true , address=104.36.82.59:4315 , piece_id=BT6AWQL6E4VICOMTLCBJA3VXWVKKCGB2FHRVTX36HZE7ODBJ5UNA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:45.893384614 (126h33m6.38581705s) [noise=true , address=104.36.82.62:4439 , piece_id=POLES4KHDMZ7NTAT6UCQSL4HSIDEZQWRWW5DUVESRR2LT3RS4NIA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:46.258630941 (126h33m6.751063377s) [noise=true , address=64.49.13.170:4720 , piece_id=6YHASHXKEEXLT2GPT6MV6WMBYWRWXMMQOQHU2OL3GFJNEMN6MYXQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:46.262308113 (126h33m6.754740553s) [noise=true , address=64.49.13.155:4919 , piece_id=3PMM67J3L3FQ5JC6FAXZ7QYGWK4QPQ74YOYCO6LDMMR2TG2WDKTQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:40:46.406037878 (126h33m6.898470307s) [noise=true , address=79.127.205.246:4279 , piece_id=2T5XZROXE3NL65TT52NYPR5NLHO4446B3SFGCPLGIF52MZ35W27A, seg_pos={0 0}, upload=111421]: Done uploading piece. err: <nil> Sep 3 20:41:44.515687444 (126h34m5.008119872s) [noise=true , address=104.36.82.55:4213 , piece_id=XBGRAYKLN47UGSK3TO4MXWCF7Z4C4YYIZVWBRU5XY7V53TU3FAVQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:44.515738904 (126h34m5.008171346s) [noise=true , address=104.36.82.55:4225 , piece_id=P2HBYO2EI7WCY4WFVWBZLESCIQKUEPSF23LANIHJVXYI7WMJXGXA, seg_pos={0 0}, upload=111421]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 139.59.144.22:48532->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:44.515764634 (126h34m5.008197061s) [noise=true , address=104.36.82.51:4062 , piece_id=2ZRRVJOOXFCIDMMST7FW6JPFXITXV26YSTR56CIVRPTFNEN6W5RQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: rpc: dial tcp 104.36.82.51:4062: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:44.515796164 (126h34m5.008228607s) [noise=true , address=104.36.82.3:4784 , piece_id=Z74VROHRBDITLC7YQFJXZIZ3IX5RHX3S7RQJJHCYSXK5AOLRZYJQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: ecclient: upload failed (node:1534abg2Jn4zTQzvBcDDZDLq2QZp6kBKY1q6zEbk2PC8mJrnq3, address:104.36.82.3:4784): protocol: write tcp 139.59.144.22:49996->104.36.82.3:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:44.515828908 (126h34m5.008261336s) [noise=true , address=77.90.31.19:4862 , piece_id=SJEOAHGJUBIKZHCRN5RYKHFP5SGQGLC7GWTOAQLTMLDNKJYFLSFQ, seg_pos={0 0}, upload=111421]: Done uploading piece. err: ecclient: upload failed (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz, address:77.90.31.19:4862): protocol: write tcp 139.59.144.22:48286->77.90.31.19:4862: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:41:44.515855391 (126h34m5.008287836s) [seg_pos={0 0}, upload=111421]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 3 20:41:44.619665071 (126h34m5.112097507s) [upload=111421]: Done uploading segments... Sep 3 20:41:44.619778462 (126h34m5.112210899s) [upload=111421]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.107802048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X......XX......X......X....X..............................] duration: 2.42232834s FAIL (2025-09-03 20:41:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.4223561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..X......XX......X......X....X.............................] duration: 2.492263649s FAIL (2025-09-03 20:41:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.492298194s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X.XXX.........XXX.XXX.XX.X........XX.....................] duration: 2.881420818s FAIL (2025-09-03 20:41:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.881452276s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..X......XX......X......X....X............................] duration: 3.243208673s FAIL (2025-09-03 20:40:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.243239146s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X.XXX.........XXX.XXX.XX.X........XX....................] duration: 2.671722498s FAIL (2025-09-03 20:40:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.67175798s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..................X.X.....] duration: 2.92195028s FAIL (2025-09-03 20:40:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Begin Sep 3 20:36:06.417290420 (126h28m26.909722862s) [seg_pos={0 0}, upload=111355]: Begin upload segment... Sep 3 20:36:06.417339513 (126h28m26.909771944s) [seg_pos={0 0}, upload=111355]: Done begin upload segment. Sep 3 20:36:06.417365702 (126h28m26.909798147s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417383305 (126h28m26.909815753s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417796664 (126h28m26.910229096s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417884882 (126h28m26.91031733s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418161134 (126h28m26.910593583s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418517778 (126h28m26.910950223s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.420479737 (126h28m26.912912189s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.423499592 (126h28m26.915932041s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425143969 (126h28m26.917576415s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425521798 (126h28m26.917954245s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425897033 (126h28m26.918329488s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.426285928 (126h28m26.9187184s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428070508 (126h28m26.92050297s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428524339 (126h28m26.920956791s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428877597 (126h28m26.92131005s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429222418 (126h28m26.921654867s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429544316 (126h28m26.921976773s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429870595 (126h28m26.922303043s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:07.000107635 (126h28m27.492540083s) [upload=111355]: Next returned nil segment Sep 3 20:36:07.000169715 (126h28m27.492602166s) [upload=111355]: Waiting for error group managing segments... Sep 3 20:36:07.175992914 (126h28m27.668425351s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.200825762 (126h28m27.693258196s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.217123039 (126h28m27.709555471s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.245138835 (126h28m27.737571287s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.252706098 (126h28m27.745138529s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.275059650 (126h28m27.767492085s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.333033834 (126h28m27.82546627s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.334202593 (126h28m27.826635036s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.353193927 (126h28m27.845626356s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.679989782 (126h28m28.172422217s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.719805088 (126h28m28.212237519s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:08.055903987 (126h28m28.548336418s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:37:05.913172399 (126h29m26.405604844s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: write tcp 139.59.144.22:45308->104.36.82.38:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913449985 (126h29m26.405882442s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: write tcp 139.59.144.22:43904->104.36.82.47:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913719223 (126h29m26.406151669s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913996555 (126h29m26.406429005s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914236157 (126h29m26.406668603s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:46828->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914293624 (126h29m26.406726074s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq, address:104.36.82.36:4648): protocol: write tcp 139.59.144.22:35890->104.36.82.36:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914358995 (126h29m26.406791444s) [seg_pos={0 0}, upload=111355]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:37:06.049747974 (126h29m26.54218041s) [upload=111355]: Done uploading segments... Sep 3 20:37:06.049782628 (126h29m26.54221506s) [upload=111355]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.921990897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..X......XX......X......X....X...........................] duration: 2.470546197s FAIL (2025-09-03 20:39:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.470651341s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..X.XXX.........XXX.XXX.XX.X........XX...................] duration: 2.792575479s FAIL (2025-09-03 20:39:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.792603218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..................X.X....] duration: 13.319369386s FAIL (2025-09-03 20:39:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Begin Sep 3 20:36:06.417290420 (126h28m26.909722862s) [seg_pos={0 0}, upload=111355]: Begin upload segment... Sep 3 20:36:06.417339513 (126h28m26.909771944s) [seg_pos={0 0}, upload=111355]: Done begin upload segment. Sep 3 20:36:06.417365702 (126h28m26.909798147s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417383305 (126h28m26.909815753s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417796664 (126h28m26.910229096s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417884882 (126h28m26.91031733s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418161134 (126h28m26.910593583s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418517778 (126h28m26.910950223s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.420479737 (126h28m26.912912189s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.423499592 (126h28m26.915932041s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425143969 (126h28m26.917576415s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425521798 (126h28m26.917954245s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425897033 (126h28m26.918329488s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.426285928 (126h28m26.9187184s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428070508 (126h28m26.92050297s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428524339 (126h28m26.920956791s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428877597 (126h28m26.92131005s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429222418 (126h28m26.921654867s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429544316 (126h28m26.921976773s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429870595 (126h28m26.922303043s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:07.000107635 (126h28m27.492540083s) [upload=111355]: Next returned nil segment Sep 3 20:36:07.000169715 (126h28m27.492602166s) [upload=111355]: Waiting for error group managing segments... Sep 3 20:36:07.175992914 (126h28m27.668425351s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.200825762 (126h28m27.693258196s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.217123039 (126h28m27.709555471s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.245138835 (126h28m27.737571287s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.252706098 (126h28m27.745138529s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.275059650 (126h28m27.767492085s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.333033834 (126h28m27.82546627s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.334202593 (126h28m27.826635036s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.353193927 (126h28m27.845626356s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.679989782 (126h28m28.172422217s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.719805088 (126h28m28.212237519s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:08.055903987 (126h28m28.548336418s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:37:05.913172399 (126h29m26.405604844s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: write tcp 139.59.144.22:45308->104.36.82.38:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913449985 (126h29m26.405882442s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: write tcp 139.59.144.22:43904->104.36.82.47:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913719223 (126h29m26.406151669s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913996555 (126h29m26.406429005s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914236157 (126h29m26.406668603s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:46828->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914293624 (126h29m26.406726074s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq, address:104.36.82.36:4648): protocol: write tcp 139.59.144.22:35890->104.36.82.36:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914358995 (126h29m26.406791444s) [seg_pos={0 0}, upload=111355]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:37:06.049747974 (126h29m26.54218041s) [upload=111355]: Done uploading segments... Sep 3 20:37:06.049782628 (126h29m26.54221506s) [upload=111355]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 13.319480049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..X......XX......X......X....X..........................] duration: 2.705828637s FAIL (2025-09-03 20:39:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.705862671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..................X.X...] duration: 4.761197469s FAIL (2025-09-03 20:39:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Begin Sep 3 20:36:06.417290420 (126h28m26.909722862s) [seg_pos={0 0}, upload=111355]: Begin upload segment... Sep 3 20:36:06.417339513 (126h28m26.909771944s) [seg_pos={0 0}, upload=111355]: Done begin upload segment. Sep 3 20:36:06.417365702 (126h28m26.909798147s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417383305 (126h28m26.909815753s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417796664 (126h28m26.910229096s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417884882 (126h28m26.91031733s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418161134 (126h28m26.910593583s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418517778 (126h28m26.910950223s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.420479737 (126h28m26.912912189s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.423499592 (126h28m26.915932041s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425143969 (126h28m26.917576415s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425521798 (126h28m26.917954245s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425897033 (126h28m26.918329488s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.426285928 (126h28m26.9187184s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428070508 (126h28m26.92050297s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428524339 (126h28m26.920956791s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428877597 (126h28m26.92131005s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429222418 (126h28m26.921654867s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429544316 (126h28m26.921976773s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429870595 (126h28m26.922303043s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:07.000107635 (126h28m27.492540083s) [upload=111355]: Next returned nil segment Sep 3 20:36:07.000169715 (126h28m27.492602166s) [upload=111355]: Waiting for error group managing segments... Sep 3 20:36:07.175992914 (126h28m27.668425351s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.200825762 (126h28m27.693258196s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.217123039 (126h28m27.709555471s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.245138835 (126h28m27.737571287s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.252706098 (126h28m27.745138529s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.275059650 (126h28m27.767492085s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.333033834 (126h28m27.82546627s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.334202593 (126h28m27.826635036s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.353193927 (126h28m27.845626356s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.679989782 (126h28m28.172422217s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.719805088 (126h28m28.212237519s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:08.055903987 (126h28m28.548336418s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:37:05.913172399 (126h29m26.405604844s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: write tcp 139.59.144.22:45308->104.36.82.38:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913449985 (126h29m26.405882442s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: write tcp 139.59.144.22:43904->104.36.82.47:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913719223 (126h29m26.406151669s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913996555 (126h29m26.406429005s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914236157 (126h29m26.406668603s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:46828->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914293624 (126h29m26.406726074s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq, address:104.36.82.36:4648): protocol: write tcp 139.59.144.22:35890->104.36.82.36:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914358995 (126h29m26.406791444s) [seg_pos={0 0}, upload=111355]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:37:06.049747974 (126h29m26.54218041s) [upload=111355]: Done uploading segments... Sep 3 20:37:06.049782628 (126h29m26.54221506s) [upload=111355]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 4.761345138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XX..X.XXX.........XXX.XXX.XX.X........XX..................] duration: 2.877163279s FAIL (2025-09-03 20:38:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.87719149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..X......XX......X......X....X.........................] duration: 2.488932923s FAIL (2025-09-03 20:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 3 20:19:59.160975149 (126h12m19.653407592s) [upload=111115]: Starting upload Sep 3 20:19:59.161046442 (126h12m19.653478887s) [upload=111115]: Done starting upload Sep 3 20:19:59.161324224 (126h12m19.653756652s) [upload=111115]: Uploading segments... Sep 3 20:19:59.161368922 (126h12m19.653801367s) [upload=111115]: Got next segment. Inline: false Sep 3 20:19:59.161402930 (126h12m19.653835373s) [upload=111115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 3 20:19:59.161407898 (126h12m19.653840341s) [upload=111115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 3 20:19:59.441173706 (126h12m19.933606138s) [seg_pos={0 0}, upload=111115]: Begin upload segment... Sep 3 20:19:59.441232055 (126h12m19.933664487s) [seg_pos={0 0}, upload=111115]: Done begin upload segment. Sep 3 20:19:59.441262305 (126h12m19.933694751s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441307709 (126h12m19.933740164s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441642451 (126h12m19.934074896s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.441895673 (126h12m19.934328125s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442080526 (126h12m19.934512972s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442437922 (126h12m19.934870369s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442450277 (126h12m19.934882729s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442684853 (126h12m19.935117298s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442916393 (126h12m19.935348839s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.442933217 (126h12m19.935365669s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443153630 (126h12m19.935586074s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443448024 (126h12m19.935880474s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.443463986 (126h12m19.935896435s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450157309 (126h12m19.942589772s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.450753288 (126h12m19.943185734s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:19:59.451199029 (126h12m19.943631482s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Uploading piece... Sep 3 20:20:00.079826177 (126h12m20.572258645s) [upload=111115]: Next returned nil segment Sep 3 20:20:00.080369319 (126h12m20.572801776s) [upload=111115]: Waiting for error group managing segments... Sep 3 20:20:00.193969186 (126h12m20.686401628s) [noise=true , address=45.91.116.12:4227 , piece_id=RZ6ATSOLTGUH3R425NVAL4HTDG5Z4YEGZJBZW7EAXDICQXSEDYPQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.201432903 (126h12m20.693865341s) [noise=true , address=77.90.31.12:4589 , piece_id=HK2GI47RAAQWMSOKTTUJINMBEM46VWBGY7OLFHJPDTEXT7JAE6RA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.206800194 (126h12m20.69923263s) [noise=true , address=45.91.116.10:4082 , piece_id=TFSFTK5ECS3M4GEOSKMWUL26Y4ALPD6R4UDN5QBTKGY5F7TDCC2A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.210342618 (126h12m20.702775047s) [noise=true , address=45.91.116.4:4939 , piece_id=PL77HHI3ST53XBYJSDFP6Y65ODEF6YNNNTAQS535W6APPGQP7G5A, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.213337891 (126h12m20.705770327s) [noise=true , address=45.91.116.5:4298 , piece_id=YFYGEMAA5CPAWEBDQTOC4PTUHTLQDJ5XFG5ON2NVMLY24JEG2FSQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.228743352 (126h12m20.72117579s) [noise=true , address=77.90.31.16:4735 , piece_id=OJRSYQMB3MBKBRZHC3NQBRRB42IO33JPOLWMRX52RH6JD6T4T6JA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.229004233 (126h12m20.72143667s) [noise=true , address=45.91.116.7:4009 , piece_id=DWSPNYG3HHK5UR7PJ4XJID3QUTSJ7N4ATBKHYE2PPHRMH47H7GEQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.235842697 (126h12m20.728275139s) [noise=true , address=45.91.116.8:4048 , piece_id=2QIRGQFNSHVLMQLVPBGZI4M6TQH6HWKWM6R43GCBDWPYQDHFXCSA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.245052412 (126h12m20.737484865s) [noise=true , address=77.90.31.12:4592 , piece_id=I3GX53G37C573VDPYBLPO4LRAIKJ4LIPUUTZ3IWFHJFQ5Q3HZGXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.247624340 (126h12m20.740056796s) [noise=true , address=45.91.116.6:4973 , piece_id=4FSCB7CVE6WBGHETQ3M2423JSFLQREIOVN6ZINIJ6CVTDB5ZIA4Q, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.254021587 (126h12m20.746454039s) [noise=true , address=45.91.116.14:4191 , piece_id=35AZHG5L6IRUCFQH3DTVGWCHQBHCW6FSLCNDYFW2FFXFOYSQJ4OA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:00.290334141 (126h12m20.782766584s) [noise=true , address=45.91.116.5:4297 , piece_id=A6UZQRCH7N2ZHNWPBNDA6V5L2MPPFISH4AS2MYXP4IVLJA4EV4BQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: <nil> Sep 3 20:20:28.859703096 (126h12m49.352135532s) [noise=true , address=104.36.82.5:4837 , piece_id=BNIFRIJNVXB6QB3BBIYPW3TIET3AH2MY4GK2AT27CAMDRYUKNAXA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 139.59.144.22:58482->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859802423 (126h12m49.352234854s) [noise=true , address=104.36.82.9:4007 , piece_id=E4RJBDZ7KEP4BJA6N5QGSXYQGOMTKO7YFQNYZPWS2KYFK5HAV5HA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 139.59.144.22:42990->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.859878688 (126h12m49.352311123s) [noise=true , address=104.36.82.10:4079 , piece_id=LIM2MKZS4OTZX2XMRRLR635LYQCJLUZUSBCB642ENHNICQ24BLTQ, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 139.59.144.22:47454->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860049221 (126h12m49.352481653s) [noise=true , address=104.36.82.6:4876 , piece_id=3Z7YIPRC76RCXBWSRTUWWY5P75YP5OFIM2U5463ZHBZBLPZJ7TAA, seg_pos={0 0}, upload=111115]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:20:28.860083452 (126h12m49.3525159s) [seg_pos={0 0}, upload=111115]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 3 20:20:28.970913369 (126h12m49.463345815s) [upload=111115]: Done uploading segments... Sep 3 20:20:28.970993100 (126h12m49.463425529s) [upload=111115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.488976156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X..................X.X..] duration: 3.2239652s FAIL (2025-09-03 20:38:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Begin Sep 3 20:36:06.417290420 (126h28m26.909722862s) [seg_pos={0 0}, upload=111355]: Begin upload segment... Sep 3 20:36:06.417339513 (126h28m26.909771944s) [seg_pos={0 0}, upload=111355]: Done begin upload segment. Sep 3 20:36:06.417365702 (126h28m26.909798147s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417383305 (126h28m26.909815753s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417796664 (126h28m26.910229096s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.417884882 (126h28m26.91031733s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418161134 (126h28m26.910593583s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.418517778 (126h28m26.910950223s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.420479737 (126h28m26.912912189s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.423499592 (126h28m26.915932041s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425143969 (126h28m26.917576415s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425521798 (126h28m26.917954245s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.425897033 (126h28m26.918329488s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.426285928 (126h28m26.9187184s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428070508 (126h28m26.92050297s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428524339 (126h28m26.920956791s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.428877597 (126h28m26.92131005s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429222418 (126h28m26.921654867s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429544316 (126h28m26.921976773s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:06.429870595 (126h28m26.922303043s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Uploading piece... Sep 3 20:36:07.000107635 (126h28m27.492540083s) [upload=111355]: Next returned nil segment Sep 3 20:36:07.000169715 (126h28m27.492602166s) [upload=111355]: Waiting for error group managing segments... Sep 3 20:36:07.175992914 (126h28m27.668425351s) [noise=true , address=79.127.219.47:4055 , piece_id=OZCN333J3ENXGWCBJLGGOYH4L6XMFFATSRHMRBG3XJSLW4KUVC7Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.200825762 (126h28m27.693258196s) [noise=true , address=104.36.82.60:4345 , piece_id=2D6ZTYVB72OFFKN7RCCXUZTIBUVCSICMEBFEYLVKDLQDT3LIZBIA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.217123039 (126h28m27.709555471s) [noise=true , address=104.36.82.23:4468 , piece_id=46BLAT2CKGMITFHHI6PTNPKS2F67NVGZ7SQHYYAA2GXTGOIXERJQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.245138835 (126h28m27.737571287s) [noise=true , address=77.90.31.14:4664 , piece_id=FNRE6QJ4SR5PSKGVGSPSZI5HLDWTTLHUBP4NJSKGULIFEJBAYNXA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.252706098 (126h28m27.745138529s) [noise=true , address=104.36.82.62:4443 , piece_id=KL44OTKMIRY3M6LJSOTHCX5PAC2YY3H7MHYFQKLSGZFUJQVGRLRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.275059650 (126h28m27.767492085s) [noise=true , address=45.91.116.8:4072 , piece_id=JWC7WWNPD75KEPIFQANKUZNYXDASWQN3KUJD33C5PVKPNIOCJBWA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.333033834 (126h28m27.82546627s) [noise=true , address=104.36.82.61:4407 , piece_id=2V7Y3ZYYF5BPZDRQNDL6BOPTPNIYQM6PQLFDYUUDFW2VDKL4ORYA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.334202593 (126h28m27.826635036s) [noise=true , address=104.36.82.12:4156 , piece_id=FHTLUH4LGPFBFHHJ6UQY6FC74QCX5VPCQMEQDIZISM3WHEMJOKRQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.353193927 (126h28m27.845626356s) [noise=true , address=104.36.82.59:4306 , piece_id=IIX4SWADSCD6NBQYZCK7WODX73C6LGJJEU46MXPYZ7UR6X4DYF2Q, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.679989782 (126h28m28.172422217s) [noise=true , address=79.127.205.246:4287 , piece_id=VGXOSIZAEY2GEG754JSAE5YZE3RPJHKBDCHMUOLWLIUYNBM3JFDQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:07.719805088 (126h28m28.212237519s) [noise=true , address=64.49.13.169:4709 , piece_id=APQIWLH7MHZUGYOJ6VHZTTTGVHDCFBCAP2Z3CNIMVSON5JSAUOKA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:36:08.055903987 (126h28m28.548336418s) [noise=true , address=45.91.116.12:4257 , piece_id=UWMW2GUSF2OPGRTIOLLNE6EMLLUUV6XAGEU6A5RVRHD7477D3PBQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: <nil> Sep 3 20:37:05.913172399 (126h29m26.405604844s) [noise=true , address=104.36.82.38:4715 , piece_id=23OF7YUA6ADYJINILKZ5C6SBOPRYE4HXRF7OWL3WIYGL3AUCN4RA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: write tcp 139.59.144.22:45308->104.36.82.38:4715: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913449985 (126h29m26.405882442s) [noise=true , address=104.36.82.47:4541 , piece_id=R2RZNI2ZBM5BBUJO3FFIAFGLGSKWDWP725KDR2VBR7YFRVEIQSSA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: write tcp 139.59.144.22:43904->104.36.82.47:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913719223 (126h29m26.406151669s) [noise=true , address=104.36.82.38:4712 , piece_id=PUDN3NBEQD3SSWAGXEW45HRUENTUPMBUNDTC7XBKYUG4Q4HYGBFA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1Ee1iVoSwRL1jpEU8Ccc7jSE7A7e9vDmHBpbLGdMrU6Uc3zCJe, address:104.36.82.38:4712): protocol: rpc: dial tcp 104.36.82.38:4712: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.913996555 (126h29m26.406429005s) [noise=true , address=104.36.82.55:4217 , piece_id=7ZWXSEXD3CQB5MCNFSKV3RZUMEMG77YD3EKC62YBXOLPIMEENTQA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914236157 (126h29m26.406668603s) [noise=true , address=104.36.82.53:4135 , piece_id=PIS5TLB3AZX7FCP2IL6WACD3PFDDZ2PFIP2GUGSSWXCLNQBCQVTA, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 139.59.144.22:46828->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914293624 (126h29m26.406726074s) [noise=true , address=104.36.82.36:4648 , piece_id=WAAKHMK4VXTBCUZBEJYA56TODINJPKYMNW5PKINUCQJPCADWCQAQ, seg_pos={0 0}, upload=111355]: Done uploading piece. err: ecclient: upload failed (node:1XHNZDY7xKZ2bss1rSdjbGP3ugSwGRwRzBSWHfXgdKKz794wMq, address:104.36.82.36:4648): protocol: write tcp 139.59.144.22:35890->104.36.82.36:4648: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 3 20:37:05.914358995 (126h29m26.406791444s) [seg_pos={0 0}, upload=111355]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Sep 3 20:37:06.049747974 (126h29m26.54218041s) [upload=111355]: Done uploading segments... Sep 3 20:37:06.049782628 (126h29m26.54221506s) [upload=111355]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.224021986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.XX..X.XXX.........XXX.XXX.XX.X........XX.................] duration: 2.826328132s FAIL (2025-09-03 20:37:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.82635651s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)