Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...................X....] duration: 1.066002133s FAIL (2025-08-19 18:08:20 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 18:05:33.459596469 (264h39m55.418693724s) [upload=275522]: Starting upload Aug 19 18:05:33.459682236 (264h39m55.418779468s) [upload=275522]: Done starting upload Aug 19 18:05:33.459785088 (264h39m55.418882321s) [upload=275522]: Uploading segments... Aug 19 18:05:33.459861817 (264h39m55.418959073s) [upload=275522]: Got next segment. Inline: true Aug 19 18:05:33.459867891 (264h39m55.41896515s) [upload=275522]: Waiting for error group managing segments... Aug 19 18:05:33.459938680 (264h39m55.419035936s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 18:05:33.459944606 (264h39m55.419041862s) [upload=275522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 18:05:33.459947990 (264h39m55.41904524s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 18:05:33.631724311 (264h39m55.590821569s) [upload=275522]: Done uploading segments... Aug 19 18:05:33.631764613 (264h39m55.590861868s) [upload=275522]: Upload finished. err: <nil> Duration:* 1.066076423s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.638135999s FAIL (2025-08-19 18:08:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.638209901s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.......................................X.............] duration: 2.399707095s FAIL (2025-08-19 18:08:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.399746124s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...................X...] duration: 1.04776856s FAIL (2025-08-19 18:08:00 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 18:05:33.459596469 (264h39m55.418693724s) [upload=275522]: Starting upload Aug 19 18:05:33.459682236 (264h39m55.418779468s) [upload=275522]: Done starting upload Aug 19 18:05:33.459785088 (264h39m55.418882321s) [upload=275522]: Uploading segments... Aug 19 18:05:33.459861817 (264h39m55.418959073s) [upload=275522]: Got next segment. Inline: true Aug 19 18:05:33.459867891 (264h39m55.41896515s) [upload=275522]: Waiting for error group managing segments... Aug 19 18:05:33.459938680 (264h39m55.419035936s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 18:05:33.459944606 (264h39m55.419041862s) [upload=275522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 18:05:33.459947990 (264h39m55.41904524s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 18:05:33.631724311 (264h39m55.590821569s) [upload=275522]: Done uploading segments... Aug 19 18:05:33.631764613 (264h39m55.590861868s) [upload=275522]: Upload finished. err: <nil> Duration:* 1.047794052s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.716934906s FAIL (2025-08-19 18:07:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.716961715s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.703367044s FAIL (2025-08-19 18:07:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.703397293s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.311760806s FAIL (2025-08-19 18:07:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.311798865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.......................................X............] duration: 2.623774063s FAIL (2025-08-19 18:06:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.623801597s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...................X..] duration: 1.256139381s FAIL (2025-08-19 18:06:46 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 18:05:33.459596469 (264h39m55.418693724s) [upload=275522]: Starting upload Aug 19 18:05:33.459682236 (264h39m55.418779468s) [upload=275522]: Done starting upload Aug 19 18:05:33.459785088 (264h39m55.418882321s) [upload=275522]: Uploading segments... Aug 19 18:05:33.459861817 (264h39m55.418959073s) [upload=275522]: Got next segment. Inline: true Aug 19 18:05:33.459867891 (264h39m55.41896515s) [upload=275522]: Waiting for error group managing segments... Aug 19 18:05:33.459938680 (264h39m55.419035936s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 18:05:33.459944606 (264h39m55.419041862s) [upload=275522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 18:05:33.459947990 (264h39m55.41904524s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 18:05:33.631724311 (264h39m55.590821569s) [upload=275522]: Done uploading segments... Aug 19 18:05:33.631764613 (264h39m55.590861868s) [upload=275522]: Upload finished. err: <nil> Duration:* 1.256173164s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 2.591824443s FAIL (2025-08-19 18:06:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.59187186s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....X......................................................] duration: 2.428011856s FAIL (2025-08-19 18:06:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.428038865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...................X.] duration: 1.249190486s FAIL (2025-08-19 18:05:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 18:05:33.459596469 (264h39m55.418693724s) [upload=275522]: Starting upload Aug 19 18:05:33.459682236 (264h39m55.418779468s) [upload=275522]: Done starting upload Aug 19 18:05:33.459785088 (264h39m55.418882321s) [upload=275522]: Uploading segments... Aug 19 18:05:33.459861817 (264h39m55.418959073s) [upload=275522]: Got next segment. Inline: true Aug 19 18:05:33.459867891 (264h39m55.41896515s) [upload=275522]: Waiting for error group managing segments... Aug 19 18:05:33.459938680 (264h39m55.419035936s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 18:05:33.459944606 (264h39m55.419041862s) [upload=275522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 18:05:33.459947990 (264h39m55.41904524s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 18:05:33.631724311 (264h39m55.590821569s) [upload=275522]: Done uploading segments... Aug 19 18:05:33.631764613 (264h39m55.590861868s) [upload=275522]: Upload finished. err: <nil> Duration:* 1.249225974s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.......................................X...........] duration: 2.330437912s FAIL (2025-08-19 18:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.330493279s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X....X.....................................................] duration: 2.308810408s FAIL (2025-08-19 18:05:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.308842532s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX...X......................................................] duration: 2.89177495s FAIL (2025-08-19 18:05:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.891798165s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 633.933094ms FAIL (2025-08-19 18:05:33 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== Aug 19 18:05:33.459596469 (264h39m55.418693724s) [upload=275522]: Starting upload Aug 19 18:05:33.459682236 (264h39m55.418779468s) [upload=275522]: Done starting upload Aug 19 18:05:33.459785088 (264h39m55.418882321s) [upload=275522]: Uploading segments... Aug 19 18:05:33.459861817 (264h39m55.418959073s) [upload=275522]: Got next segment. Inline: true Aug 19 18:05:33.459867891 (264h39m55.41896515s) [upload=275522]: Waiting for error group managing segments... Aug 19 18:05:33.459938680 (264h39m55.419035936s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 18:05:33.459944606 (264h39m55.419041862s) [upload=275522]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 18:05:33.459947990 (264h39m55.41904524s) [upload=275522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 18:05:33.631724311 (264h39m55.590821569s) [upload=275522]: Done uploading segments... Aug 19 18:05:33.631764613 (264h39m55.590861868s) [upload=275522]: Upload finished. err: <nil> Duration:* 634.034192ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.......................................X..........] duration: 2.493468592s FAIL (2025-08-19 18:05:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.493555284s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................................X.........] duration: 2.432005152s FAIL (2025-08-19 18:04:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.432042516s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 867.952776ms FAIL (2025-08-19 18:04:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 867.989555ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X....X....................................................] duration: 2.488003615s FAIL (2025-08-19 18:04:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.488036234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................................X........] duration: 2.443062459s FAIL (2025-08-19 18:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.443130692s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX...X.....................................................] duration: 2.786573389s FAIL (2025-08-19 18:04:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786594752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X....X...................................................] duration: 2.425787309s FAIL (2025-08-19 18:04:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.425804709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................................X.......] duration: 2.418434491s FAIL (2025-08-19 18:04:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.41847723s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 970.827067ms FAIL (2025-08-19 18:03:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 970.863452ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX...X....................................................] duration: 2.67648331s FAIL (2025-08-19 18:03:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.676543944s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 2.51398521s FAIL (2025-08-19 18:03:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.514058353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.......................................X......] duration: 2.722945446s FAIL (2025-08-19 18:03:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.722983037s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 2.351463279s FAIL (2025-08-19 18:03:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.351495686s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX...X...................................................] duration: 2.852177084s FAIL (2025-08-19 18:02:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.852212273s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.......................................X.....] duration: 2.410904294s FAIL (2025-08-19 18:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.410992367s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.453792413s FAIL (2025-08-19 18:02:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 1.453827337s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 2.461734709s FAIL (2025-08-19 18:02:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.461824358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.......................................X....] duration: 2.674077582s FAIL (2025-08-19 18:02:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.674176128s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 973.669192ms FAIL (2025-08-19 18:01:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 973.699469ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....XX...X..................................................] duration: 2.860719003s FAIL (2025-08-19 18:01:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.860743811s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 2.342109048s FAIL (2025-08-19 18:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.342152341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 966.579025ms FAIL (2025-08-19 18:01:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 966.617804ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.......................................X...] duration: 2.420915495s FAIL (2025-08-19 18:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.420955492s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....XX...X.................................................] duration: 2.894554077s FAIL (2025-08-19 18:00:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.894574769s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 922.563715ms FAIL (2025-08-19 18:00:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 922.594693ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 2.354048626s FAIL (2025-08-19 18:00:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.354077743s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XX...X................................................] duration: 2.933585935s FAIL (2025-08-19 18:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.933616306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.......................................X..] duration: 2.474888719s FAIL (2025-08-19 18:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.475003686s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 2.413708196s FAIL (2025-08-19 18:00:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.413735718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 854.866578ms FAIL (2025-08-19 17:59:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 854.888432ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.......................................X.] duration: 2.51225788s FAIL (2025-08-19 17:59:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 19 17:58:50.067322221 (264h33m12.026419467s) [upload=275405]: Starting upload Aug 19 17:58:50.067378466 (264h33m12.026475712s) [upload=275405]: Done starting upload Aug 19 17:58:50.067501149 (264h33m12.026598399s) [upload=275405]: Uploading segments... Aug 19 17:58:50.067753585 (264h33m12.026850868s) [upload=275405]: Got next segment. Inline: false Aug 19 17:58:50.067807416 (264h33m12.026904877s) [upload=275405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:58:50.067815146 (264h33m12.026912402s) [upload=275405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:58:50.160160613 (264h33m12.119257861s) [upload=275405]: Next segment err: metaclient: Too Many Requests Aug 19 17:58:50.160182599 (264h33m12.119279845s) [upload=275405]: Done uploading segments... Aug 19 17:58:50.160199783 (264h33m12.119297029s) [upload=275405]: Upload finished. err: metaclient: Too Many Requests Duration: 2.512314566s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 2.502601702s FAIL (2025-08-19 17:59:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 24:18.796252873 (263h58m40.755350129s) [upload=274814]: Starting upload Aug 19 17:24:18.796347370 (263h58m40.755444624s) [upload=274814]: Done starting upload Aug 19 17:24:18.796478659 (263h58m40.755575892s) [upload=274814]: Uploading segments... Aug 19 17:24:18.796704449 (263h58m40.755801694s) [upload=274814]: Got next segment. Inline: false Aug 19 17:24:18.796799006 (263h58m40.75589627s) [upload=274814]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:24:18.796832679 (263h58m40.755929931s) [upload=274814]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 19 17:24:18.905028499 (263h58m40.864125743s) [seg_pos={0 0}, upload=274814]: Begin upload segment... Aug 19 17:24:18.905074585 (263h58m40.864171839s) [seg_pos={0 0}, upload=274814]: Done begin upload segment. Aug 19 17:24:18.905113736 (263h58m40.864210985s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.905609601 (263h58m40.864706903s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906070869 (263h58m40.865168157s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906133921 (263h58m40.865231194s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906540584 (263h58m40.865637882s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906619237 (263h58m40.865716488s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906970419 (263h58m40.866067685s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.906994358 (263h58m40.86609161s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907418339 (263h58m40.866515607s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.907930651 (263h58m40.867027911s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908333186 (263h58m40.867430448s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.908715422 (263h58m40.867812691s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909117232 (263h58m40.8682145s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909535649 (263h58m40.868632906s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.909676095 (263h58m40.868773349s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:18.910066094 (263h58m40.869163331s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Uploading piece... Aug 19 17:24:19.556905276 (263h58m41.516002533s) [upload=274814]: Next returned nil segment Aug 19 17:24:19.556928380 (263h58m41.516025629s) [upload=274814]: Waiting for error group managing segments... Aug 19 17:24:19.690521166 (263h58m41.649618415s) [noise=true , address=104.36.82.4:4801 , piece_id=4OATQAEVNLEIBD74KR553JTFDY4QPQ3IC5D2P56CDM2IXV74MYKQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.698202242 (263h58m41.657299488s) [noise=true , address=104.36.82.14:4222 , piece_id=JMKXEPNJKEAUA222YBHHTIRAYS5MSYOLLG7EC62TNF5X54CN3JKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.701130779 (263h58m41.660228018s) [noise=true , address=104.36.82.10:4077 , piece_id=QQ7YOYYNH4T2KSOFUHQGUQIYDJFLA4F5MAXW3GZKWY4I6DNJVKKA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.705038145 (263h58m41.664135382s) [noise=true , address=104.36.82.11:4115 , piece_id=FPFK5X6TVCATGNMD65PLQAVQV33YHOYDYDJ43TQ4H4DTQ75UUAAA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.765827758 (263h58m41.724925005s) [noise=true , address=104.36.82.9:4006 , piece_id=N4WDXRART2WVXQQHCRKXZTOF5UZ7N63MFH7YY2LXCOWV5TZZIA7A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772068748 (263h58m41.731165982s) [noise=true , address=104.36.82.8:4970 , piece_id=H5OVBWUFVN7CZ4V3A22JJFSLX7XBRPC3ULQC5WWZNKLRBZJGVV5A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.772538630 (263h58m41.731635861s) [noise=true , address=104.36.82.2:4730 , piece_id=TMUZPRRPMKRDXYX7ZEHNEEZVRDHCUASZAWWI32H3V4QLDWJM2VAQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.776560345 (263h58m41.73565758s) [noise=true , address=104.36.82.3:4765 , piece_id=5CI7TKCTBS4OS4L5ITGWB337BIZMPS3YFFP2ZXITRGWKP5PGKJ6Q, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:19.783565902 (263h58m41.742663141s) [noise=true , address=104.36.82.12:4149 , piece_id=4GVDL73G23CMUF4KQHFO6LX7K34HVJU2ITYBXUQY32GL2COTTRNA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: <nil> Aug 19 17:24:48.529091459 (263h59m10.488188697s) [noise=true , address=45.91.116.5:4298 , piece_id=4KOLHXZVIG2D7NZZCQH3DBNPXPOOSKPBFJLVH5XATBEMXV3UUU2A, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: write tcp 139.59.144.22:40920->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529154410 (263h59m10.488251646s) [noise=true , address=77.90.31.11:4554 , piece_id=S2OJCW5AWFJLNOPYZZGF5II5VBMWYFGAOP7IN64G7DCWGMPDS5LQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 139.59.144.22:54122->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529418008 (263h59m10.488515254s) [noise=true , address=45.91.116.9:4262 , piece_id=LZSUVOQZALD7RMZTIYRHHDFYMR5K7XGZADRETZGYAI666MHRPRRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529484471 (263h59m10.488581717s) [noise=true , address=45.91.116.11:4117 , piece_id=VPBSZED66LZF67ILRDUMWBQIOVDE6AVY6VR6LBHKAZGOR5GAMZVA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:44648->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529537064 (263h59m10.48863431s) [noise=true , address=45.91.116.6:4975 , piece_id=MAOQD4M3ZMRDP3HY4KHFJXKYJZQ74P44I2HSQHO2MPKKGYVKO4WQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 139.59.144.22:46304->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529592979 (263h59m10.488690214s) [noise=true , address=77.90.31.17:4772 , piece_id=N4LZL5NJYD5SI3W7T2ETKBQUB76MRIZYGUPCN3MUGQNK7XGBMKRQ, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: read tcp 139.59.144.22:40914->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529645811 (263h59m10.488743047s) [noise=true , address=77.90.31.12:4589 , piece_id=WSFQ3PD2XCPQCDCFAQV4UQR5MTJ4RKYCQQ3YV66Q2MQRWF3DHGBA, seg_pos={0 0}, upload=274814]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:46612->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 19 17:24:48.529680055 (263h59m10.488777303s) [seg_pos={0 0}, upload=274814]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Aug 19 17:24:48.633821394 (263h59m10.592918627s) [upload=274814]: Done uploading segments... Aug 19 17:24:48.633856611 (263h59m10.592953867s) [upload=274814]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.502630543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 853.653823ms FAIL (2025-08-19 17:59:18 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/1K") ======LOGS====== Aug 19 17:52:00.969965685 (264h26m22.929062931s) [upload=275286]: Starting upload Aug 19 17:52:00.970066099 (264h26m22.929163344s) [upload=275286]: Done starting upload Aug 19 17:52:00.970199065 (264h26m22.929296298s) [upload=275286]: Uploading segments... Aug 19 17:52:00.970246480 (264h26m22.929343732s) [upload=275286]: Got next segment. Inline: true Aug 19 17:52:00.970252146 (264h26m22.929349391s) [upload=275286]: Waiting for error group managing segments... Aug 19 17:52:00.970298490 (264h26m22.929395735s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 19 17:52:00.970303940 (264h26m22.929401186s) [upload=275286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 19 17:52:00.970307395 (264h26m22.92940464s) [upload=275286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 19 17:52:01.293132685 (264h26m23.252229931s) [upload=275286]: Done uploading segments... Aug 19 17:52:01.293180230 (264h26m23.252277524s) [upload=275286]: Upload finished. err: <nil> Duration:* 853.703686ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XX...X...............................................] duration: 2.868089554s FAIL (2025-08-19 17:59:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.86818491s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)