Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 2.328305251s FAIL (2025-08-19 17:52:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.328420431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.573319934s FAIL (2025-08-19 17:52:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.573354853s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 947.38521ms FAIL (2025-08-19 17:52:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 947.418329ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX...X......................................] duration: 2.854316653s FAIL (2025-08-19 17:52:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.854408897s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.88398523s FAIL (2025-08-19 17:52:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.884018372s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 873.554478ms FAIL (2025-08-19 17:52:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 873.606398ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.555367228s FAIL (2025-08-19 17:51:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.555421426s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 2.446537815s FAIL (2025-08-19 17:51:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.446571493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XX...X.....................................] duration: 2.930047182s FAIL (2025-08-19 17:51:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.930065513s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 2.533833579s FAIL (2025-08-19 17:50:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.53391428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.642006526s FAIL (2025-08-19 17:50:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.642060123s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XX...X....................................] duration: 2.922413027s FAIL (2025-08-19 17:50:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.922483537s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.724254821s FAIL (2025-08-19 17:50:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.724391064s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XX...X...................................] duration: 2.73716417s FAIL (2025-08-19 17:49:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.737196827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 2.486400454s FAIL (2025-08-19 17:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.486435225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.482620936s FAIL (2025-08-19 17:49:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.482695758s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 2.486144663s FAIL (2025-08-19 17:48:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.486247233s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XX...X..................................] duration: 2.801169482s FAIL (2025-08-19 17:48:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801203376s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.456511225s FAIL (2025-08-19 17:48:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.456541901s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 2.405477949s FAIL (2025-08-19 17:48:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.405507941s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XX...X.................................] duration: 3.193927049s FAIL (2025-08-19 17:48:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.19397331s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XX...X................................] duration: 2.741264806s FAIL (2025-08-19 17:47:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.741290945s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.391181394s FAIL (2025-08-19 17:47:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.391232012s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 2.406695297s FAIL (2025-08-19 17:47:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.406736569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 2.409170148s FAIL (2025-08-19 17:46:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.409200337s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XX...X...............................] duration: 2.731570503s FAIL (2025-08-19 17:46:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.731602632s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XX...X..............................] duration: 2.579931443s FAIL (2025-08-19 17:46:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.57995773s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.490956134s FAIL (2025-08-19 17:46:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.490989315s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 2.508509705s FAIL (2025-08-19 17:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.508540887s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XX...X.............................] duration: 2.810490077s FAIL (2025-08-19 17:45:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.810517446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 990.070168ms FAIL (2025-08-19 17:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 990.176971ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.572291687s FAIL (2025-08-19 17:45:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.572400812s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XX...X............................] duration: 2.758427017s FAIL (2025-08-19 17:44:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.758501528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 2.487674962s FAIL (2025-08-19 17:44:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.487782168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XX...X...........................] duration: 2.713521139s FAIL (2025-08-19 17:44:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.71356791s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 2.361921426s FAIL (2025-08-19 17:44:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.361956509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 968.913571ms FAIL (2025-08-19 17:44:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 968.947276ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.431155703s FAIL (2025-08-19 17:43:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.431189818s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 848.158123ms FAIL (2025-08-19 17:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 848.187977ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 2.715908258s FAIL (2025-08-19 17:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.715942317s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XX...X..........................] duration: 2.749378489s FAIL (2025-08-19 17:43:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.749451493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 2.484820529s FAIL (2025-08-19 17:43:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.484860049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.613203245s FAIL (2025-08-19 17:42:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.61325176s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.932743492s FAIL (2025-08-19 17:42:26 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.932780417s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 1.101511282s FAIL (2025-08-19 17:42:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 1.101544473s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 904.545503ms FAIL (2025-08-19 17:42:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after 587eec1da72ded826391011bb1d204e86a565ba2b8e785848a00714f40d08995 != before 85ce0bca23ef88ba016af8b122357e6137391de16b9f8d81cfbb24236b3c9f3d ======LOGS====== Aug 19 17:02:32.318054162 (263h36m54.277151407s) [upload=274444]: Starting upload Aug 19 17:02:32.318099520 (263h36m54.277196775s) [upload=274444]: Done starting upload Aug 19 17:02:32.318164631 (263h36m54.277261866s) [upload=274444]: Uploading segments... Aug 19 17:02:32.318647445 (263h36m54.277744716s) [upload=274444]: Got next segment. Inline: true Aug 19 17:02:32.318668562 (263h36m54.277765808s) [upload=274444]: Waiting for error group managing segments... Aug 19 17:02:32.318799219 (263h36m54.277896467s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:02:32.318808240 (263h36m54.277905485s) [upload=274444]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:02:32.318814215 (263h36m54.27791146s) [upload=274444]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:02:32.482202347 (263h36m54.441299603s) [upload=274444]: Done uploading segments... Aug 19 17:02:32.482253168 (263h36m54.441350414s) [upload=274444]: Upload finished. err: <nil> Duration:* 904.619182ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XX...X.........................] duration: 2.742809144s FAIL (2025-08-19 17:42:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.742842242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 2.348955347s FAIL (2025-08-19 17:41:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.34898648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.361842747s FAIL (2025-08-19 17:41:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 19 17:27:25.515663440 (264h1m47.474760677s) [upload=274865]: Starting upload Aug 19 17:27:25.515707538 (264h1m47.474804777s) [upload=274865]: Done starting upload Aug 19 17:27:25.515993870 (264h1m47.475091101s) [upload=274865]: Uploading segments... Aug 19 17:27:25.516039634 (264h1m47.475136888s) [upload=274865]: Got next segment. Inline: false Aug 19 17:27:25.516070700 (264h1m47.475167945s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:27:25.516075278 (264h1m47.475172523s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:27:25.715650676 (264h1m47.674747924s) [seg_pos={0 0}, upload=274865]: Begin upload segment... Aug 19 17:27:25.715728311 (264h1m47.674825566s) [seg_pos={0 0}, upload=274865]: Done begin upload segment. Aug 19 17:27:25.715777484 (264h1m47.674874734s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716199496 (264h1m47.675296749s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.716719202 (264h1m47.675816455s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717247803 (264h1m47.67634506s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.717626748 (264h1m47.676723994s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718199088 (264h1m47.677296358s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.718467709 (264h1m47.677564958s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719020632 (264h1m47.678117888s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719450592 (264h1m47.678547845s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.719946518 (264h1m47.679043754s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720313828 (264h1m47.679411079s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720555945 (264h1m47.679653197s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.720919899 (264h1m47.680017155s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721292007 (264h1m47.680389257s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.721650528 (264h1m47.680747781s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722090154 (264h1m47.681187408s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722464569 (264h1m47.681561819s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:25.722819189 (264h1m47.681916447s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Uploading piece... Aug 19 17:27:26.288376322 (264h1m48.247473565s) [upload=274865]: Next returned nil segment Aug 19 17:27:26.288399254 (264h1m48.247496509s) [upload=274865]: Waiting for error group managing segments... Aug 19 17:27:26.459148397 (264h1m48.418245646s) [noise=true , address=104.36.82.36:4635 , piece_id=VWHGN7HY6YHIK4NCWGE4RJAMSCSJ6XRVPJ7VQ5Y26T4GE34AN32Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.466397421 (264h1m48.425494667s) [noise=true , address=79.127.219.47:4062 , piece_id=NVSLSVMTL2QYADXPN63KTIRNBN2CNDJ3QD2GJ5E4OFWH4WFFTSMA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.467575941 (264h1m48.426673185s) [noise=true , address=104.36.82.8:4982 , piece_id=RMNRRD2IIYXQE4YDVJHJAUALAOYCIEY4W675LDO4RGOHZ5ELY5DQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.468734053 (264h1m48.427831293s) [noise=true , address=104.36.82.17:4337 , piece_id=27RZJRAZLQ4F4MHIWALLMFBFIR2Q6KYBLJBG6F6U3Q6E272HSZSA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490127163 (264h1m48.449224406s) [noise=true , address=104.36.82.4:4812 , piece_id=ADCKM554SNWIPZDYKWWB24CBZZT2JA2SSGNXLXLVSKWFFRT4W63A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.490729203 (264h1m48.44982644s) [noise=true , address=104.36.82.5:4871 , piece_id=MZUUMU2EAJ4RMHWCE273AKGIBHKNHGI5ANUTAIECZKJUOTZGALXA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.491412933 (264h1m48.450510167s) [noise=true , address=104.36.82.35:4585 , piece_id=ICRLM5NLWMRSO5TSFODM7I6THLQXFQFPFEXFYUKQ5CGKW75NXC7Q, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.494297444 (264h1m48.453394695s) [noise=true , address=104.36.82.44:4882 , piece_id=HIEDFFBFRG2UIPYW6XMCUXIEBPXBBWK7YVKQQNVF2QII6F6N7K5A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.563406042 (264h1m48.522503273s) [noise=true , address=104.36.82.6:4904 , piece_id=5YBECPZAXAH32PBLHTYLSG7YDZWBB2QCCQ7IM5WXCXM3VE7I6RAA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.758454081 (264h1m48.71755132s) [noise=true , address=45.91.116.11:4134 , piece_id=Y7AXI2WUUUYDORALQZGBTNTDLKW6BWOWSOHUXKQ6Z2PYRIYSMC6A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.759039972 (264h1m48.718137218s) [noise=true , address=45.91.116.9:4287 , piece_id=FLPURSIT4MZYHPQABZ6L3276S5LY6WWMPVTRCZPFWPW7KWTSJ3VA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.823407099 (264h1m48.782504345s) [noise=true , address=77.90.31.15:4729 , piece_id=6I6JZJ3EEMN5YQ375PG53R26YCWRYWPU3Q62FGI5OGWHLV5LIYCA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.842199042 (264h1m48.801296279s) [noise=true , address=45.91.116.8:4057 , piece_id=NYZK7ETAHQ2RLUBXWE3VSUH4VY6Q4BMFUE7GGM7SGR2L6QKJISLA, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911529072 (264h1m48.870626308s) [noise=true , address=77.90.31.16:4750 , piece_id=N2FN3EE7PSM6T3FYTP7PXMROMTTTGJ36KFUAU6CJMDOQ4U4EXPOQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: <nil> Aug 19 17:27:26.911591812 (264h1m48.870689057s) [seg_pos={0 0}, upload=274865]: Segment reached optimal threshold of 14 pieces. Aug 19 17:27:26.911823059 (264h1m48.870920298s) [noise=true , address=45.91.116.12:4256 , piece_id=HR3FSWPQHPNSHMAZ4F3EC3OVGIS26HNJHYZVNG776XL7QOTLYE3A, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.911912918 (264h1m48.871010158s) [noise=true , address=77.90.31.17:4774 , piece_id=SFC5CJIGIQUIGN374JTQRUZDJHXWXWBMVYCHFX7T4ZPYECFSVGTQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mZTv2q5zT1Za4DTcdFbUwWs1897vv9GXuLqPcnY3FaxTdGv3Y): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912035513 (264h1m48.87113276s) [noise=true , address=77.90.31.10:4531 , piece_id=5NA627TOFDZT32BLUAFMVTIZ45VRWBLA2AFMFXUKLP2MCM2LJ6CQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912052105 (264h1m48.871149342s) [noise=true , address=45.91.116.6:4985 , piece_id=WLSDFAWL2IFWO3JPYQEZ22CNIVHEXWTVLWOJ7RSQP74PUD6JMBSQ, seg_pos={0 0}, upload=274865]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: expected piece hash; context canceled; EOF Aug 19 17:27:26.912076852 (264h1m48.871174092s) [seg_pos={0 0}, upload=274865]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 19 17:27:26.912157722 (264h1m48.871254974s) [upload=274865]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 19 17:27:26.912163879 (264h1m48.871261129s) [upload=274865]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:27:27.236815163 (264h1m49.195912401s) [upload=274865]: Done uploading segments... Aug 19 17:27:27.236836191 (264h1m49.19593343s) [upload=274865]: Upload finished. err: <nil> Duration: 2.361889596s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XX...X........................] duration: 2.972366194s FAIL (2025-08-19 17:41:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.972411766s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)