Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...X.............................................X....X....] duration: 2.737505739s FAIL (2025-08-19 17:27:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.737555138s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...XX...XX.......................................XX...X.....] duration: 2.956315574s FAIL (2025-08-19 17:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.956342717s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........X.X...................................] duration: 875.033859ms FAIL (2025-08-19 17:27:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 875.066703ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXX.....................................................X] duration: 2.315380219s FAIL (2025-08-19 17:27:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.315413145s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 2.546569585s FAIL (2025-08-19 17:27:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.546611089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X..........X.X..................................] duration: 831.452343ms FAIL (2025-08-19 17:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 831.498241ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...X.............................................X....X...] duration: 3.660643147s FAIL (2025-08-19 17:27:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 3.660677955s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....XX...XX.......................................XX...X....] duration: 2.86017791s FAIL (2025-08-19 17:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.860205231s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....XX...XX.......................................XX...X...] duration: 2.865773914s FAIL (2025-08-19 17:26:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.865797348s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........X.X.................................] duration: 878.101408ms FAIL (2025-08-19 17:26:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 878.130459ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 2.600048389s FAIL (2025-08-19 17:26:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.600080185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 2.344026383s FAIL (2025-08-19 17:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.344057765s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 2.472115924s FAIL (2025-08-19 17:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.472146251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X.............................................X....X..] duration: 2.618103417s FAIL (2025-08-19 17:26:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.618298604s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XX...XX.......................................XX...X..] duration: 2.603750293s FAIL (2025-08-19 17:25:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.60384497s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........X.X................................] duration: 878.353322ms FAIL (2025-08-19 17:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 878.387461ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 3.700870708s FAIL (2025-08-19 17:25:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.70089747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 9.783633003s FAIL (2025-08-19 17:25:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 9.783657008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....XXX....................................................] duration: 9.240439939s FAIL (2025-08-19 17:25:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 9.240488604s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 35.086542678s FAIL (2025-08-19 17:25:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 35.086583627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....XXX...................................................] duration: 34.37167762s FAIL (2025-08-19 17:24:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 34.371708828s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........X.X...............................] duration: 1.271327742s FAIL (2025-08-19 17:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.271361488s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XX...XX.......................................XX...X.] duration: 3.720846194s FAIL (2025-08-19 17:24:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.720861681s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...X.............................................X....X.] duration: 2.973111541s FAIL (2025-08-19 17:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.973136479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...X.............................................X....X] duration: 30.10723276s FAIL (2025-08-19 17:24:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.107263706s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........XX...XX.......................................XX...X] duration: 30.001372311s FAIL (2025-08-19 17:24:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0013973s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X..........X.X..............................] duration: 905.80395ms FAIL (2025-08-19 17:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 905.835389ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 2.60767118s FAIL (2025-08-19 17:23:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.607707492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........XX...XX.......................................XX...] duration: 2.977246251s FAIL (2025-08-19 17:23:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.9773016s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X..........X.X.............................] duration: 982.981514ms FAIL (2025-08-19 17:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 983.012743ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X....XXX..................................................] duration: 5.159353035s FAIL (2025-08-19 17:23:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 5.159382222s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...X.............................................X....] duration: 2.337585559s FAIL (2025-08-19 17:23:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== g_pos={0 0}, upload=274746]: Done begin upload segment. Aug 19 17:20:12.517299055 (263h54m34.476396316s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517352367 (263h54m34.476449614s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517906151 (263h54m34.477003405s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.518970487 (263h54m34.478067758s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.527756455 (263h54m34.486853712s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.529479293 (263h54m34.488576552s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.530812572 (263h54m34.489909829s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.531631939 (263h54m34.490729186s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533142469 (263h54m34.492239724s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533701029 (263h54m34.492798278s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.535438042 (263h54m34.494535291s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536023701 (263h54m34.495120958s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536066814 (263h54m34.495164073s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536561518 (263h54m34.495658771s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.541679757 (263h54m34.500777058s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.547878488 (263h54m34.506975744s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:13.159450228 (263h54m35.118547485s) [upload=274746]: Next returned nil segment Aug 19 17:20:13.159475337 (263h54m35.118572594s) [upload=274746]: Waiting for error group managing segments... Aug 19 17:20:13.299805425 (263h54m35.25890291s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.306959062 (263h54m35.266056309s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.313612241 (263h54m35.272709491s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.318313058 (263h54m35.277410304s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.345273172 (263h54m35.304370417s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.349055852 (263h54m35.308153098s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:42.130306460 (263h55m4.089403715s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:55248->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130362353 (263h55m4.089459602s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 139.59.144.22:44410->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130399253 (263h55m4.08949651s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130459138 (263h55m4.08955641s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130544367 (263h55m4.089641612s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 139.59.144.22:60086->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130599561 (263h55m4.089696818s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 139.59.144.22:51998->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130736453 (263h55m4.089833698s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:36312->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130784968 (263h55m4.089882214s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:41740->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130820582 (263h55m4.089917839s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:52394->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130894850 (263h55m4.089992096s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:52162->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130941956 (263h55m4.090039211s) [seg_pos={0 0}, upload=274746]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:20:42.233818698 (263h55m4.192915948s) [upload=274746]: Done uploading segments... Aug 19 17:20:42.233869039 (263h55m4.192966285s) [upload=274746]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.337619742s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........XX...XX.......................................XX..] duration: 2.592531435s FAIL (2025-08-19 17:22:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.592564279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 2.474353539s FAIL (2025-08-19 17:22:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.474384215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X..........X.X............................] duration: 1.262332089s FAIL (2025-08-19 17:22:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.262372545s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...X.............................................X...] duration: 2.337718306s FAIL (2025-08-19 17:22:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== g_pos={0 0}, upload=274746]: Done begin upload segment. Aug 19 17:20:12.517299055 (263h54m34.476396316s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517352367 (263h54m34.476449614s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517906151 (263h54m34.477003405s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.518970487 (263h54m34.478067758s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.527756455 (263h54m34.486853712s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.529479293 (263h54m34.488576552s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.530812572 (263h54m34.489909829s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.531631939 (263h54m34.490729186s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533142469 (263h54m34.492239724s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533701029 (263h54m34.492798278s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.535438042 (263h54m34.494535291s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536023701 (263h54m34.495120958s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536066814 (263h54m34.495164073s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536561518 (263h54m34.495658771s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.541679757 (263h54m34.500777058s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.547878488 (263h54m34.506975744s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:13.159450228 (263h54m35.118547485s) [upload=274746]: Next returned nil segment Aug 19 17:20:13.159475337 (263h54m35.118572594s) [upload=274746]: Waiting for error group managing segments... Aug 19 17:20:13.299805425 (263h54m35.25890291s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.306959062 (263h54m35.266056309s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.313612241 (263h54m35.272709491s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.318313058 (263h54m35.277410304s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.345273172 (263h54m35.304370417s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.349055852 (263h54m35.308153098s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:42.130306460 (263h55m4.089403715s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:55248->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130362353 (263h55m4.089459602s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 139.59.144.22:44410->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130399253 (263h55m4.08949651s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130459138 (263h55m4.08955641s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130544367 (263h55m4.089641612s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 139.59.144.22:60086->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130599561 (263h55m4.089696818s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 139.59.144.22:51998->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130736453 (263h55m4.089833698s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:36312->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130784968 (263h55m4.089882214s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:41740->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130820582 (263h55m4.089917839s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:52394->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130894850 (263h55m4.089992096s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:52162->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130941956 (263h55m4.090039211s) [seg_pos={0 0}, upload=274746]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:20:42.233818698 (263h55m4.192915948s) [upload=274746]: Done uploading segments... Aug 19 17:20:42.233869039 (263h55m4.192966285s) [upload=274746]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 2.337771283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X....XXX.................................................] duration: 2.770273391s FAIL (2025-08-19 17:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 2.770364607s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...X.............................................X..] duration: 3.803198137s FAIL (2025-08-19 17:21:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== g_pos={0 0}, upload=274746]: Done begin upload segment. Aug 19 17:20:12.517299055 (263h54m34.476396316s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517352367 (263h54m34.476449614s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517906151 (263h54m34.477003405s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.518970487 (263h54m34.478067758s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.527756455 (263h54m34.486853712s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.529479293 (263h54m34.488576552s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.530812572 (263h54m34.489909829s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.531631939 (263h54m34.490729186s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533142469 (263h54m34.492239724s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533701029 (263h54m34.492798278s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.535438042 (263h54m34.494535291s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536023701 (263h54m34.495120958s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536066814 (263h54m34.495164073s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536561518 (263h54m34.495658771s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.541679757 (263h54m34.500777058s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.547878488 (263h54m34.506975744s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:13.159450228 (263h54m35.118547485s) [upload=274746]: Next returned nil segment Aug 19 17:20:13.159475337 (263h54m35.118572594s) [upload=274746]: Waiting for error group managing segments... Aug 19 17:20:13.299805425 (263h54m35.25890291s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.306959062 (263h54m35.266056309s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.313612241 (263h54m35.272709491s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.318313058 (263h54m35.277410304s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.345273172 (263h54m35.304370417s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.349055852 (263h54m35.308153098s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:42.130306460 (263h55m4.089403715s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:55248->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130362353 (263h55m4.089459602s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 139.59.144.22:44410->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130399253 (263h55m4.08949651s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130459138 (263h55m4.08955641s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130544367 (263h55m4.089641612s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 139.59.144.22:60086->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130599561 (263h55m4.089696818s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 139.59.144.22:51998->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130736453 (263h55m4.089833698s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:36312->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130784968 (263h55m4.089882214s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:41740->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130820582 (263h55m4.089917839s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:52394->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130894850 (263h55m4.089992096s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:52162->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130941956 (263h55m4.090039211s) [seg_pos={0 0}, upload=274746]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:20:42.233818698 (263h55m4.192915948s) [upload=274746]: Done uploading segments... Aug 19 17:20:42.233869039 (263h55m4.192966285s) [upload=274746]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 3.803233962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 3.624171297s FAIL (2025-08-19 17:21:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.624203042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........XX...XX.......................................XX.] duration: 3.393234198s FAIL (2025-08-19 17:21:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393928796s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X....XXX................................................] duration: 7.958652868s FAIL (2025-08-19 17:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 7.958691103s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................X..........X.X...........................] duration: 1.793884921s FAIL (2025-08-19 17:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.793918546s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...X.............................................X.] duration: 6.271160458s FAIL (2025-08-19 17:21:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== g_pos={0 0}, upload=274746]: Done begin upload segment. Aug 19 17:20:12.517299055 (263h54m34.476396316s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517352367 (263h54m34.476449614s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517906151 (263h54m34.477003405s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.518970487 (263h54m34.478067758s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.527756455 (263h54m34.486853712s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.529479293 (263h54m34.488576552s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.530812572 (263h54m34.489909829s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.531631939 (263h54m34.490729186s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533142469 (263h54m34.492239724s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533701029 (263h54m34.492798278s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.535438042 (263h54m34.494535291s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536023701 (263h54m34.495120958s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536066814 (263h54m34.495164073s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536561518 (263h54m34.495658771s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.541679757 (263h54m34.500777058s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.547878488 (263h54m34.506975744s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:13.159450228 (263h54m35.118547485s) [upload=274746]: Next returned nil segment Aug 19 17:20:13.159475337 (263h54m35.118572594s) [upload=274746]: Waiting for error group managing segments... Aug 19 17:20:13.299805425 (263h54m35.25890291s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.306959062 (263h54m35.266056309s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.313612241 (263h54m35.272709491s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.318313058 (263h54m35.277410304s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.345273172 (263h54m35.304370417s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.349055852 (263h54m35.308153098s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:42.130306460 (263h55m4.089403715s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:55248->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130362353 (263h55m4.089459602s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 139.59.144.22:44410->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130399253 (263h55m4.08949651s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130459138 (263h55m4.08955641s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130544367 (263h55m4.089641612s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 139.59.144.22:60086->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130599561 (263h55m4.089696818s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 139.59.144.22:51998->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130736453 (263h55m4.089833698s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:36312->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130784968 (263h55m4.089882214s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:41740->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130820582 (263h55m4.089917839s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:52394->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130894850 (263h55m4.089992096s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:52162->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130941956 (263h55m4.090039211s) [seg_pos={0 0}, upload=274746]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:20:42.233818698 (263h55m4.192915948s) [upload=274746]: Done uploading segments... Aug 19 17:20:42.233869039 (263h55m4.192966285s) [upload=274746]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 6.271191497s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............XX...XX.......................................XX] duration: 30.001411517s FAIL (2025-08-19 17:21:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001450181s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 10.957137949s FAIL (2025-08-19 17:21:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== em_SegmentBegin Aug 19 16:48:38.279467993 (263h23m0.238565238s) [seg_pos={0 0}, upload=274203]: Begin upload segment... Aug 19 16:48:38.279531640 (263h23m0.238628887s) [seg_pos={0 0}, upload=274203]: Done begin upload segment. Aug 19 16:48:38.279555099 (263h23m0.238652374s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279566523 (263h23m0.238663761s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279956583 (263h23m0.239053829s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.279998838 (263h23m0.239096076s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280258538 (263h23m0.239355783s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.280557381 (263h23m0.239654627s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.281981520 (263h23m0.24107877s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.282838036 (263h23m0.24193528s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.283671458 (263h23m0.242768706s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.284459061 (263h23m0.24355631s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285473551 (263h23m0.244570796s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.285679398 (263h23m0.244776647s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286178644 (263h23m0.245275892s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.286796608 (263h23m0.245893854s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287303197 (263h23m0.246400463s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.287792627 (263h23m0.246889883s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289110882 (263h23m0.248208138s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.289455715 (263h23m0.248552961s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Uploading piece... Aug 19 16:48:38.863904774 (263h23m0.823002015s) [upload=274203]: Next returned nil segment Aug 19 16:48:38.863961427 (263h23m0.823058685s) [upload=274203]: Waiting for error group managing segments... Aug 19 16:48:39.011763544 (263h23m0.970860776s) [noise=true , address=104.36.82.60:4337 , piece_id=HCJ5H77G6BGAHJIGAQZDEYJMDAAZMZAGFG2SAZJ7S7XSMPQRIWZQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.030272112 (263h23m0.989369349s) [noise=true , address=104.36.82.49:4992 , piece_id=Q3NUNKL3VKYCQES5AIWSZJBBFIQNFBNSEHI7PSG62B2R7UV42EKQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.032518957 (263h23m0.991616195s) [noise=true , address=104.36.82.61:4391 , piece_id=HX4HHCVW246OXM6EZ2VLCNYFP7KAIFINDA24NVL7KOPUGTBFUTMQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.056066657 (263h23m1.015163901s) [noise=true , address=104.36.82.8:4994 , piece_id=EURI4IMKABAZCER5B5PQRKH3MB7CFO3KAGGJKX3Y2TTLVJNKDZHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.059095795 (263h23m1.018193033s) [noise=true , address=104.36.82.39:4754 , piece_id=IR2M7RCDM5CYH4AVFAQ4TRPCCI5INCBOEHUPWYF76HZZ5EYFMJXA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.076902010 (263h23m1.035999242s) [noise=true , address=104.36.82.10:4111 , piece_id=ATC33YMG6WLA6MCLGRN6ZO5Q45LMVZUDAPMI355TN5JS7IAJXJZA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.109152270 (263h23m1.068249508s) [noise=true , address=104.36.82.51:4082 , piece_id=FBJ4YUUNQBXBCYR6B6DQ5B5IZ4MCUJSORFSL3Q2SFWMODQL43A3A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.142579296 (263h23m1.101676529s) [noise=true , address=104.36.82.23:4474 , piece_id=7EAO4SRXFEHCROC2F5MMM536SIL75NQJPOQ7ACGOOBVMQNPHH55Q, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.263057564 (263h23m1.222154804s) [noise=true , address=79.127.220.101:4075 , piece_id=66HWCIAPXQURLMJH6CHRWGZC755KEDHTPEDIPOORU6PM4FMBYD2A, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.515983542 (263h23m1.475080776s) [noise=true , address=66.23.193.175:4563 , piece_id=QK77ZRVRXFELZJUTA333VOVPOWMNFHQTONBKTJBSMFWIOVT7MOFQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:48:39.696554842 (263h23m1.655652081s) [noise=true , address=79.127.205.245:4234 , piece_id=DNMLMTMY6P7XFF6QPEW3QBAYXJHVOENBW3TYZHFPSZTF7RWJFLGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:34.793806523 (263h23m56.752903755s) [noise=true , address=45.91.116.6:4001 , piece_id=JRWAJTIHT3H3UECW66A4WEUKR64ED7PIZR3CRYQZODSLVE4PLIOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: <nil> Aug 19 16:49:37.805134831 (263h23m59.764232086s) [noise=true , address=45.91.116.12:4246 , piece_id=5LABDLDMNNYGVL3ONU4TBJEGFWU6AHTI7UFTDNWZG2QZEOU7LYHQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: read tcp 139.59.144.22:40908->45.91.116.12:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805214837 (263h23m59.764312074s) [noise=true , address=45.91.116.11:4133 , piece_id=L74QZ5Y6UKLAE754FBWNY7Y2DASDRKH6ELTRXNTPD6LCPZFLHYNQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 139.59.144.22:46702->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805277910 (263h23m59.764375161s) [noise=true , address=77.90.31.18:4835 , piece_id=BQQFPUHAGWZZ6NF7HUWUPNPUVELJT3E33RWYNPXZI234DQLNHPOA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: read tcp 139.59.144.22:33438->77.90.31.18:4835: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805307837 (263h23m59.764405085s) [noise=true , address=45.91.116.8:4071 , piece_id=R4X4II35JUXYFTAXBJYKELLYIFJTW6BE462JK7LMOLRHHYUYFSOQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12eSNywSRC4ouKS4qHnP5pFowBVMNQFwC1HHcLtrfbJDijevXEi, address:45.91.116.8:4071): protocol: context deadline exceeded; context deadline exceeded Aug 19 16:49:37.805379847 (263h23m59.764477093s) [noise=true , address=45.91.116.4:4969 , piece_id=MFBPMIQF7ZF2ULUAWAEMT3IAUC4J4HWN2INRRAWX5WPHRFTGVMFA, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx, address:45.91.116.4:4969): protocol: read tcp 139.59.144.22:59670->45.91.116.4:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 16:49:37.805478905 (263h23m59.764576159s) [noise=true , address=45.91.116.7:4034 , piece_id=66245FTV3D7SBCD7O7KCSBNYRWPY7PB6V2GH3YWSOUS26DRD2JGQ, seg_pos={0 0}, upload=274203]: Done uploading piece. err: ecclient: upload failed (node:1paGTvsyFrNyTYAi86hWdnNWoL2aP1u1kdtYQKsiszrB8FR6Qp, address:45.91.116.7:4034): protocol: write tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; close tcp 139.59.144.22:40892->45.91.116.7:4034: use of closed network connection; context deadline exceeded Aug 19 16:49:37.805517563 (263h23m59.76461482s) [seg_pos={0 0}, upload=274203]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 19 16:49:37.922784779 (263h23m59.881882018s) [upload=274203]: Done uploading segments... Aug 19 16:49:37.922821061 (263h23m59.881918298s) [upload=274203]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 10.95726559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X....XXX...............................................] duration: 18.430289915s FAIL (2025-08-19 17:21:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 4c3b2e414af11ff09a815a194fd226cfd3dc38d15fa6f177dab55bbb7fb802b4 != before f70300871679a92a8dfacd07db047024ef634f7dc4aea76f18a10d4bde16fe62 ======LOGS====== Aug 19 16:43:02.310369726 (263h17m24.269466979s) [upload=274103]: Starting upload Aug 19 16:43:02.310445033 (263h17m24.269542267s) [upload=274103]: Done starting upload Aug 19 16:43:02.310561626 (263h17m24.269658864s) [upload=274103]: Uploading segments... Aug 19 16:43:02.310779926 (263h17m24.26987716s) [upload=274103]: Got next segment. Inline: false Aug 19 16:43:02.310829275 (263h17m24.269926509s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 16:43:02.310835808 (263h17m24.269933042s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 16:43:02.618955394 (263h17m24.57805268s) [seg_pos={0 0}, upload=274103]: Begin upload segment... Aug 19 16:43:02.619074653 (263h17m24.578171902s) [seg_pos={0 0}, upload=274103]: Done begin upload segment. Aug 19 16:43:02.619132825 (263h17m24.578230081s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620004777 (263h17m24.579102036s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.620842420 (263h17m24.579939679s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.621245327 (263h17m24.580342601s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622076112 (263h17m24.581173368s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.622163484 (263h17m24.58126074s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.623927954 (263h17m24.583025215s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.624758263 (263h17m24.583855734s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625186644 (263h17m24.5842839s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.625643360 (263h17m24.584740619s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626120950 (263h17m24.585218201s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626263936 (263h17m24.585361192s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.626623884 (263h17m24.585721143s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627086178 (263h17m24.586183431s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627287573 (263h17m24.586384833s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.627593102 (263h17m24.586690362s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628035641 (263h17m24.587132898s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:02.628473250 (263h17m24.587570509s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Uploading piece... Aug 19 16:43:03.228496521 (263h17m25.187593772s) [upload=274103]: Next returned nil segment Aug 19 16:43:03.228522000 (263h17m25.187619251s) [upload=274103]: Waiting for error group managing segments... Aug 19 16:43:03.413214363 (263h17m25.372311628s) [noise=true , address=45.91.116.7:4026 , piece_id=4BDYPQCFYBXDC243U23XB54XDPCZAPSUIBC2YFLL5L7O5YXXMJQA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414290016 (263h17m25.373387278s) [noise=true , address=77.90.31.15:4716 , piece_id=65RA2FWYURC5JLRJAZT363NUMWP4YCAPKK6XGSZLQZ46EIRFF32Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.414910387 (263h17m25.374007622s) [noise=true , address=45.91.116.8:4052 , piece_id=QVJU4LJTE37PODV3WX4TQDRUYVQLWCTBVTTDDSYXKKK7VEOJ63WA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.418840027 (263h17m25.377937268s) [noise=true , address=77.90.31.14:4672 , piece_id=22NXVHNHXXXVEJKECMKBBD3VYRYXMWRUKVPPLFA2PLIVQCJDQGJQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.422322660 (263h17m25.381419894s) [noise=true , address=104.36.82.61:4385 , piece_id=U26OL23EV7I6JNH7EXXDSQFGGCB6CSR4RHEPRKQCPCGT7QQTM56Q, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.424039623 (263h17m25.383136874s) [noise=true , address=45.91.116.14:4214 , piece_id=BX6MAM6SLV2D2CIOMHTZGOF5WUUHKDKZOKJ4FQUYKCIZE3XIZJHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.433795942 (263h17m25.392893195s) [noise=true , address=77.90.31.12:4596 , piece_id=IQ2WTC64EVLAH2NC3SR75KFUD3BAEBD56YQNM7EKCS5VRPVERCGA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.442914168 (263h17m25.402011424s) [noise=true , address=77.90.31.10:4537 , piece_id=GHWHXIQJQ3GYUZDMHVLO7HLRUTBN7AO2SFYRBRLTZPN6J2EJGSZA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.454675734 (263h17m25.413772979s) [noise=true , address=45.91.116.5:4328 , piece_id=OP5MTRU6UOCHJJMLYXEEOSPQPY3P65JY4EFKWCJGZILO723XR6LQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.476502593 (263h17m25.43559985s) [noise=true , address=104.36.82.5:4843 , piece_id=S46M2KMRECPQBIMSEGXEWJSMKC2PUV5U7MNSUTXL5TH352WCZHFQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.482151715 (263h17m25.441248971s) [noise=true , address=104.36.82.11:4133 , piece_id=XE7LR53TYSYFKI7CC5JJSET442W5GUG4BZUY6CFAP7LNJFKXRJIQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.493919518 (263h17m25.45301676s) [noise=true , address=104.36.82.45:4935 , piece_id=UQ7O57SFTTVPDKI5OKGBFDRXSCENK4VO3LG5DA5KMCNOXBDIVFEQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.623789365 (263h17m25.582886635s) [noise=true , address=77.90.31.13:4629 , piece_id=HL2QOW3YAQODYKF7MXFVWHSKWPOV26BPE4SG5ITM4J22JJ64BT7A, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742780904 (263h17m25.701878145s) [noise=true , address=77.90.31.19:4872 , piece_id=OZCZF4VFLMNWAWMS6JJK6RSHQKMYSS7VDYWDGGN7FV46XGWLNTTQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: <nil> Aug 19 16:43:03.742818476 (263h17m25.701915723s) [seg_pos={0 0}, upload=274103]: Segment reached optimal threshold of 14 pieces. Aug 19 16:43:03.743151641 (263h17m25.702248887s) [noise=true , address=104.36.82.14:4235 , piece_id=N2LBPED4R5W3NJGYGQ6ULTNELAAUCEX7VS7VPOE2IJUTJGDALOHQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743327396 (263h17m25.702424642s) [noise=true , address=104.36.82.62:4444 , piece_id=H5W7BYC3C5P67CL5HYB5SRXSQ445Q7FDVFHRJ4VDC6TOC4SZB5VQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pcTUmp7XFBSnNeop7QdWdJN6WpqNHU6b8rb7PGZhczs1CLjB7): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743464915 (263h17m25.702562166s) [noise=true , address=45.91.116.10:4105 , piece_id=5Z22SFUZSLNZEGGIBQWS2ZFKMRDHWNFJ474MHMS5A3RT2C2MMPRA, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743732273 (263h17m25.702829533s) [noise=true , address=79.127.205.235:4687 , piece_id=ABDUBD4A5I65JCHTGRIBCZKSR7D2II7SOV7G5FVTIPZT2QGNFCLQ, seg_pos={0 0}, upload=274103]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13Z1si8NCPYQAEsqy63rspU3hYbWsTEEHweBmoDzoL4mo6PQTx): protocol: expected piece hash; context canceled; EOF Aug 19 16:43:03.743793086 (263h17m25.702890337s) [seg_pos={0 0}, upload=274103]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 16:43:03.743905594 (263h17m25.703002845s) [upload=274103]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 16:43:03.743913204 (263h17m25.703010453s) [upload=274103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 16:43:03.970350027 (263h17m25.929447276s) [upload=274103]: Done uploading segments... Aug 19 16:43:03.970370784 (263h17m25.929468026s) [upload=274103]: Upload finished. err: <nil> Duration: 18.430321269s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX...XX.......................................X] duration: 30.014062627s FAIL (2025-08-19 17:20:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.014087808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...X.............................................X] duration: 30.106039397s FAIL (2025-08-19 17:20:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== g_pos={0 0}, upload=274746]: Done begin upload segment. Aug 19 17:20:12.517299055 (263h54m34.476396316s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517352367 (263h54m34.476449614s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.517906151 (263h54m34.477003405s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.518970487 (263h54m34.478067758s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.527756455 (263h54m34.486853712s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.529479293 (263h54m34.488576552s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.530812572 (263h54m34.489909829s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.531631939 (263h54m34.490729186s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533142469 (263h54m34.492239724s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.533701029 (263h54m34.492798278s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.535438042 (263h54m34.494535291s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536023701 (263h54m34.495120958s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536066814 (263h54m34.495164073s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.536561518 (263h54m34.495658771s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.541679757 (263h54m34.500777058s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:12.547878488 (263h54m34.506975744s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Uploading piece... Aug 19 17:20:13.159450228 (263h54m35.118547485s) [upload=274746]: Next returned nil segment Aug 19 17:20:13.159475337 (263h54m35.118572594s) [upload=274746]: Waiting for error group managing segments... Aug 19 17:20:13.299805425 (263h54m35.25890291s) [noise=true , address=104.36.82.8:4972 , piece_id=DH67N2LQPWLROGROLQZWEGN4C5Q35RZ6B43EPPYWCXJTYN55573A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.306959062 (263h54m35.266056309s) [noise=true , address=104.36.82.11:4113 , piece_id=GSPLRV7JZLOIFUDGT42CEGSJLPPVE6RI3ULPMEKQZHHHKQKFUG2Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.313612241 (263h54m35.272709491s) [noise=true , address=104.36.82.4:4801 , piece_id=EY7G6NC357VBGR5UN6GV2ODK4IKWGG2AIJGKK3B54CLRMR4NYY7Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.318313058 (263h54m35.277410304s) [noise=true , address=104.36.82.10:4077 , piece_id=Y7G6LNDNZWAVLMFX3KFCV2UUD42ZLZ6MCPVFP4KYSMGIGV7I6OZQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.345273172 (263h54m35.304370417s) [noise=true , address=104.36.82.6:4874 , piece_id=XYCSLDHWJKVO4C75AWSZL7XUF3O62ICBQORSX7N3XQDU3EYVRLTQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:13.349055852 (263h54m35.308153098s) [noise=true , address=104.36.82.2:4731 , piece_id=IC47WQZTK5LVXVMIDACFJTGPA6CBBNO2SLM7DYCV6LBQFUAVSGSQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: <nil> Aug 19 17:20:42.130306460 (263h55m4.089403715s) [noise=true , address=77.90.31.19:4842 , piece_id=XHP7W6IZIUXO5F54V7BT7TN5T44GJRQ56YXABGWTYEKNDMDT2DHA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: read tcp 139.59.144.22:55248->77.90.31.19:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130362353 (263h55m4.089459602s) [noise=true , address=45.91.116.10:4081 , piece_id=HPE6533UIU66RZOLECLGZC5JV6ETWHQBURJXFTJYFHOP3PSRKA3A, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, address:45.91.116.10:4081): protocol: write tcp 139.59.144.22:44410->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130399253 (263h55m4.08949651s) [noise=true , address=45.91.116.7:4011 , piece_id=C3KXDQUZUEUE4XEYXN32ZJDI2DQVFOT6XZYRUBEDE3AFKHXFC2BA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, address:45.91.116.7:4011): protocol: rpc: dial tcp 45.91.116.7:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130459138 (263h55m4.08955641s) [noise=true , address=45.91.116.10:4082 , piece_id=3FF2GZVVQ42DYUIHCZYH5PZC7HDGLD2LTESGFWY3BNVKIIYSYRRQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130544367 (263h55m4.089641612s) [noise=true , address=45.91.116.13:4153 , piece_id=364OGQKRKHVQBNSZ2NZIV34GJKJ2BAHDEPVASTBDSUKJLT43S24Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: read tcp 139.59.144.22:60086->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130599561 (263h55m4.089696818s) [noise=true , address=45.91.116.6:4975 , piece_id=IACMLJOJY2IABUKIGAMWR3J2UMXGUMG6FJ5D7ULFNO4RNB67KSHQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 139.59.144.22:51998->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130736453 (263h55m4.089833698s) [noise=true , address=77.90.31.15:4699 , piece_id=IAMSK7MVJIN6NFOZ6FONNDOFXZK3JMYFJM6ZXD4ANU5TMTT7A6LA, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: read tcp 139.59.144.22:36312->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130784968 (263h55m4.089882214s) [noise=true , address=77.90.31.14:4663 , piece_id=QME4YHBDJJ3NP6QGV5UFWJBNJCRJ7O2Q2XJ4OPYRQEINTC7UW7FQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 139.59.144.22:41740->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130820582 (263h55m4.089917839s) [noise=true , address=45.91.116.12:4227 , piece_id=B2EYANPCS5AUGGH4BWEJHRTMXZGUHP4EVJWRTCDDTCOZZLT3AY3Q, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 139.59.144.22:52394->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130894850 (263h55m4.089992096s) [noise=true , address=77.90.31.16:4734 , piece_id=YAS3HAEUDFOXYMAKL3GVQTHV547Q7WYZGNJVTGWGQF6TMBXBOEVQ, seg_pos={0 0}, upload=274746]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: read tcp 139.59.144.22:52162->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:20:42.130941956 (263h55m4.090039211s) [seg_pos={0 0}, upload=274746]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Aug 19 17:20:42.233818698 (263h55m4.192915948s) [upload=274746]: Done uploading segments... Aug 19 17:20:42.233869039 (263h55m4.192966285s) [upload=274746]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.106071221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................X..........X.X..........................] duration: 865.494317ms FAIL (2025-08-19 17:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 865.53904ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..........X.X.........................] duration: 945.54604ms FAIL (2025-08-19 17:20:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 945.699797ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)