Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.337939396s FAIL (2025-08-17 05:08:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.337987699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.306404535s FAIL (2025-08-17 05:08:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.30643739s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.X........X......................] duration: 1.064502692s FAIL (2025-08-17 05:08:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.064559706s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.........................X.............X......] duration: 2.389577498s FAIL (2025-08-17 05:08:12 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.389612468s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 2.938931251s FAIL (2025-08-17 05:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.938953963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.368053994s FAIL (2025-08-17 05:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.36808583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.........................X.............X.....] duration: 2.544565378s FAIL (2025-08-17 05:07:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.544597823s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.275978745s FAIL (2025-08-17 05:07:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.276031689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XXXXX......................................................] duration: 2.904206764s FAIL (2025-08-17 05:07:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.904231945s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.X........X.....................] duration: 1.074070307s FAIL (2025-08-17 05:07:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.074101362s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 2.394488327s FAIL (2025-08-17 05:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.394518871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.424387198s FAIL (2025-08-17 05:06:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.42441439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.........................X.............X....] duration: 2.412225356s FAIL (2025-08-17 05:06:50 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.412255085s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XXXXX.....................................................] duration: 2.784911711s FAIL (2025-08-17 05:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.785160287s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.724967721s FAIL (2025-08-17 05:06:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.724998274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X.X........X....................] duration: 1.04964004s FAIL (2025-08-17 05:06:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.049665763s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 2.41486283s FAIL (2025-08-17 05:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.414913937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.........................X.............X...] duration: 2.555859237s FAIL (2025-08-17 05:05:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.555896708s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 2.511732479s FAIL (2025-08-17 05:05:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.511769534s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XXXXX....................................................] duration: 2.638970373s FAIL (2025-08-17 05:05:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.638996163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 3.122093917s FAIL (2025-08-17 05:05:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 3.122124864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.622161063s FAIL (2025-08-17 05:05:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.62220075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.X........X...................] duration: 836.391451ms FAIL (2025-08-17 05:05:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 836.427438ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 2.413429373s FAIL (2025-08-17 05:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.413485343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.X........X..................] duration: 811.957127ms FAIL (2025-08-17 05:04:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 811.980647ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 3.213678741s FAIL (2025-08-17 05:04:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.213700659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 2.782841179s FAIL (2025-08-17 05:04:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.782877133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.........................X.............X..] duration: 2.483969213s FAIL (2025-08-17 05:04:52 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.484001614s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XXXXX...................................................] duration: 2.72780248s FAIL (2025-08-17 05:04:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.727827134s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.X........X.................] duration: 847.579242ms FAIL (2025-08-17 05:04:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 847.606652ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 2.529878229s FAIL (2025-08-17 05:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.529921105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 2.62085426s FAIL (2025-08-17 05:04:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.620908505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 2.481254213s FAIL (2025-08-17 05:03:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.48128688s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 3.009098802s FAIL (2025-08-17 05:03:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.009125278s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XXXXX..................................................] duration: 2.612931704s FAIL (2025-08-17 05:03:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.612980051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................................X.X........X................] duration: 963.856676ms FAIL (2025-08-17 05:03:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 963.890834ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.........................X.............X.] duration: 2.417808726s FAIL (2025-08-17 05:03:40 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 2.41783864s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 2.681926493s FAIL (2025-08-17 05:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.681957707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XXXXX.................................................] duration: 2.852892938s FAIL (2025-08-17 05:03:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.852921368s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 3.813958137s FAIL (2025-08-17 05:03:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.813989771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.........................X.............X] duration: 1.74525714s FAIL (2025-08-17 05:03:15 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 17 05:03:13.602511700 (203h37m35.561608946s) [upload=211959]: Starting upload Aug 17 05:03:13.602579829 (203h37m35.561677075s) [upload=211959]: Done starting upload Aug 17 05:03:13.602710980 (203h37m35.561808227s) [upload=211959]: Uploading segments... Aug 17 05:03:13.603262465 (203h37m35.562359752s) [upload=211959]: Got next segment. Inline: false Aug 17 05:03:13.603337276 (203h37m35.562434511s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 05:03:13.603343808 (203h37m35.562441054s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 05:03:13.799853325 (203h37m35.758950577s) [seg_pos={0 0}, upload=211959]: Begin upload segment... Aug 17 05:03:13.800036220 (203h37m35.759133474s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.800177632 (203h37m35.759274878s) [seg_pos={0 0}, upload=211959]: Done begin upload segment. Aug 17 05:03:13.800197764 (203h37m35.759295015s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801258837 (203h37m35.76035607s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801397102 (203h37m35.760494351s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.801990298 (203h37m35.761087551s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.802697918 (203h37m35.761795169s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803128784 (203h37m35.762226035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.803486952 (203h37m35.762584198s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804350699 (203h37m35.76344795s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.804699270 (203h37m35.763796515s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805076918 (203h37m35.764174172s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.805457355 (203h37m35.764554606s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807167900 (203h37m35.766265156s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.807742402 (203h37m35.766839651s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808118131 (203h37m35.767215401s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808544794 (203h37m35.767642045s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.808856024 (203h37m35.767953276s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:13.810151413 (203h37m35.769248662s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Uploading piece... Aug 17 05:03:14.362743471 (203h37m36.32184071s) [upload=211959]: Next returned nil segment Aug 17 05:03:14.362762027 (203h37m36.321859274s) [upload=211959]: Waiting for error group managing segments... Aug 17 05:03:14.508461037 (203h37m36.467558284s) [noise=true , address=104.36.82.16:4308 , piece_id=7UZYKU6QEJVJIRVCZ4QHNZCREASFLZKYHFFZDHUOUV4SDOYOGV6A, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.511434934 (203h37m36.470532178s) [noise=true , address=77.90.31.14:4676 , piece_id=UVEJOV4L6N6XK7SO4Y34F4D2V5PM5TDFNXWK3NPWX54QF76S4NDA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522280687 (203h37m36.481377927s) [noise=true , address=45.91.116.6:4981 , piece_id=6CU7QZ5WFYCSNJEV66VWPB7AQU3I5HOFZ3C2ABMCWK2NAXB5C24Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.522473561 (203h37m36.481570806s) [noise=true , address=45.91.116.14:4203 , piece_id=WJTUBKR4XWNR4AM4PJNNIQUMZNDML6SUH2YTDQJIR3OLEVFVGPJA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.525492742 (203h37m36.484589981s) [noise=true , address=104.36.82.58:4276 , piece_id=2YKRD6EESPYAFGENOKXIQ56ZEKCP4KOX6LHYZZGSXBNO2CTSBWEQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.526788397 (203h37m36.485885632s) [noise=true , address=45.91.116.10:4103 , piece_id=HVL7DUDJZU75K4HXVEC4VHSY6CDLO4RPBX4IJDKXXAEFLU3OZWLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.527513729 (203h37m36.486610965s) [noise=true , address=77.90.31.11:4580 , piece_id=EUBM2QZBJW5ARJCDWCQXN357DQJAYGYFSJCS32X5LZU3MYIS47FA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.530297757 (203h37m36.489394988s) [noise=true , address=77.90.31.16:4757 , piece_id=2ZZBHLMYXJISW4U6UJL3QOSNJXLR2QCWMHC76O4IXLLN7CX2DFUQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.531686830 (203h37m36.490784089s) [noise=true , address=77.90.31.10:4545 , piece_id=5XOPWNEWQBNYSGJ26RAUMMRTRQ2EDVYFCPMCMZ3XCS7M4FU77TZA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.539828385 (203h37m36.498925618s) [noise=true , address=77.90.31.21:4899 , piece_id=5KTNSD35RSMCMLYZBFRF7OVJQEM5LWO5QJZW3N56LGWYAYXZ43VQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.557167794 (203h37m36.516265035s) [noise=true , address=104.36.82.5:4866 , piece_id=CS4BGNYPULI3THDJXS3YMKL4DCB7BKKWZLKQCXYKH4GATV3QSWBA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.563369778 (203h37m36.52246703s) [noise=true , address=77.90.31.18:4811 , piece_id=3IFVP5JALFTI7LMOB6Z2I7AM76AXMUEMXN3LNEIDI33X7T2T6DLQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.567463984 (203h37m36.52656123s) [noise=true , address=104.36.82.59:4325 , piece_id=7EQFGGJAASXBUZTKMYO6EVTZFO7KLPA2XL6VDZQBFQOSDHYALPPQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685114757 (203h37m36.644211993s) [noise=true , address=104.36.82.23:4464 , piece_id=372GUNFP52QXR7EOT6YJHFKOIVEKQUJQZFM7R3AXD4Q6AQHBZ5NA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: <nil> Aug 17 05:03:14.685164773 (203h37m36.644262018s) [seg_pos={0 0}, upload=211959]: Segment reached optimal threshold of 14 pieces. Aug 17 05:03:14.685471845 (203h37m36.64456909s) [noise=true , address=104.36.82.60:4371 , piece_id=76GG3IT2FKXUQWKCKZFFVYD453U2L572O6XTJY6XF6TZD3EPZZXQ, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KJP5DHmfnSFQF3Q74GzMUvcAz9zh16td4pnsqHAwToiM4D1vd): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685585241 (203h37m36.644682487s) [noise=true , address=79.127.220.101:4063 , piece_id=XHMZRVZ75MULVF3K52QMPYLDEV57Q2KLAP4I6V5HDFO5KXI5QW4Q, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CAoTJPUeWPgEvSCtaBQqVK15E8jF448WGHs4yAfP6GH2mtC1x): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685605870 (203h37m36.644703119s) [noise=true , address=66.23.193.175:4561 , piece_id=UNZOW37VOG7JAEAG3ERK4TJQ3ZPFBK55TK3YK7RG4254MZ6QEJTA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bYAiLD5oLcD72fz1TE9nmNmmszSLXkzKMc2Wc56pDKc119cu3): protocol: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45824->66.23.193.175:4561: use of closed network connection Aug 17 05:03:14.685664837 (203h37m36.644762081s) [noise=true , address=79.127.201.215:4353 , piece_id=LUV57WW5M462PKQ5RRXV54FEWJCNH3NZKE3TOO2DZIL55W4O3XHA, seg_pos={0 0}, upload=211959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227Uk4K8hmXrae9npcgjziCNhwfdfdDPdgEryAy5beiFaSmd1W): protocol: expected piece hash; context canceled; EOF Aug 17 05:03:14.685688107 (203h37m36.644785342s) [seg_pos={0 0}, upload=211959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 05:03:14.685790677 (203h37m36.644887913s) [upload=211959]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 05:03:14.685796312 (203h37m36.644893557s) [upload=211959]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 05:03:14.842181821 (203h37m36.801279077s) [upload=211959]: Done uploading segments... Aug 17 05:03:14.842207356 (203h37m36.801304612s) [upload=211959]: Upload finished. err: <nil> Duration: 1.745363479s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XXXXX................................................] duration: 2.995639826s FAIL (2025-08-17 05:03:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.99567285s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 2.527982413s FAIL (2025-08-17 05:02:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.528015105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........................X.............] duration: 2.468992037s FAIL (2025-08-17 05:02:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:52:48.490540025 (203h27m10.449637277s) [upload=211774]: Starting upload Aug 17 04:52:48.490604066 (203h27m10.449701325s) [upload=211774]: Done starting upload Aug 17 04:52:48.490733030 (203h27m10.449830268s) [upload=211774]: Uploading segments... Aug 17 04:52:48.490895113 (203h27m10.449992358s) [upload=211774]: Got next segment. Inline: false Aug 17 04:52:48.490944376 (203h27m10.450041721s) [upload=211774]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:48.490952125 (203h27m10.450049373s) [upload=211774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:52:48.694225741 (203h27m10.653322987s) [seg_pos={0 0}, upload=211774]: Begin upload segment... Aug 17 04:52:48.694287146 (203h27m10.653384386s) [seg_pos={0 0}, upload=211774]: Done begin upload segment. Aug 17 04:52:48.694324474 (203h27m10.653421723s) [noise=true , address=77.90.31.11:4562 , piece_id=7UCJ2NOZELQIZAI4VTWTHAZYNELMWKV2PDERTVY4YGG4KKOWXWHQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694369283 (203h27m10.653466547s) [noise=true , address=77.90.31.18:4824 , piece_id=OAZDEBEEPYEAKQWPF7KGP4YTR5PVOHD4XO5RSLPJ653IXFJTVCDA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694778226 (203h27m10.653875472s) [noise=true , address=79.127.201.215:4360 , piece_id=AHN3MJ6MKAKQ7ALTI5Z642BCZLITRLRPY4IIA2N3DE5RMP3RTGGQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694866587 (203h27m10.653963831s) [noise=true , address=104.36.82.39:4738 , piece_id=ZIJ4NHPJY5WWE46LEZDUCULNAAWR363RJUT2DBM25RC4YEHY47YQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695133229 (203h27m10.654230476s) [noise=true , address=104.36.82.21:4508 , piece_id=AJ2ZSCAXMJB7HMHTSXUDQZJBZPCLGEBYUC2IEGTEGTB2BGHNCJCQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695323197 (203h27m10.654420443s) [noise=true , address=104.36.82.5:4866 , piece_id=SK4UJRQUOQZQVQI77ZBUPZ6BOKI3WNDPZHR5QDUF34OOKENI5EGQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695511156 (203h27m10.654608403s) [noise=true , address=104.36.82.23:4472 , piece_id=7BR3ZF7MIB5K3GE5OBUHWHK7QTE3LJIHCN5B7DZLPCD5G7FNNYRA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695658635 (203h27m10.654755879s) [noise=true , address=104.36.82.4:4818 , piece_id=QWKWGTQASLFES4NHZDQAMZYGSMU2T5MC7LHWVT7YFBKX4JAOKWBQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695902387 (203h27m10.654999635s) [noise=true , address=45.91.116.10:4098 , piece_id=7ZFFIBNOJ4GLKSZPKAUGKCVC2I7W5C3FQAXYYWXIIL7O7OARVBKQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696122792 (203h27m10.655220037s) [noise=true , address=77.90.31.21:4884 , piece_id=43HVGGMFGGUSWZ3P6YRZXRHOHGUM2JS727HMHKNZ4V5RQYZSUVRQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696279899 (203h27m10.655377147s) [noise=true , address=104.36.82.44:4904 , piece_id=6S5LDTNJP226KVAB2I6PNFZDKHKH4IQD5MRRBBBZVQKQYWUDPNEQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696603169 (203h27m10.655700414s) [noise=true , address=104.36.82.59:4325 , piece_id=VUO4P5OLQRTK3BIC5TRA37TKAFV6KOMA25PVDIWHBKV7NO32ZMPQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696649532 (203h27m10.65574678s) [noise=true , address=104.36.82.40:4962 , piece_id=6KUC3GJQ4MCNQW4665FNPUSR3SSKP7XKSO2M63XZJW7UBKCW5YPA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697180139 (203h27m10.656277387s) [noise=true , address=45.91.116.14:4203 , piece_id=ZJEVJ4DRTQQSAU2NUTEBPXAMUS3RAZ34D7B2ZPAL26D22XEPKYAQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697194676 (203h27m10.65629196s) [noise=true , address=77.90.31.19:4858 , piece_id=3IFWCPGWHNPOWK32CGAYVH7DRRKL3KY57UBJ4RJNJEGKXS34OBKA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697581706 (203h27m10.656678954s) [noise=true , address=104.36.82.60:4342 , piece_id=A36WWC2UFQKGYZHKOTNVRF7EHESDRIH7V2LLKLVLLNPTUAXLNQ5A, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697725008 (203h27m10.656822252s) [noise=true , address=45.91.116.5:4320 , piece_id=IKUCNDOTLMU4RDFQW5BHULQQBATM4UYNR7E5RXPFDUZQ53OXV37Q, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697903656 (203h27m10.657000947s) [noise=true , address=104.36.82.16:4308 , piece_id=XTNKSQXKFAYLKO5ASHMS2XWHR2ICMTGHMZTJGI47UUBA4DUYTWSA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:49.240699808 (203h27m11.199797041s) [upload=211774]: Next returned nil segment Aug 17 04:52:49.240756486 (203h27m11.199853836s) [upload=211774]: Waiting for error group managing segments... Aug 17 04:52:49.399120274 (203h27m11.35821751s) [noise=true , address=104.36.82.39:4738 , piece_id=ZIJ4NHPJY5WWE46LEZDUCULNAAWR363RJUT2DBM25RC4YEHY47YQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399144035 (203h27m11.358241275s) [noise=true , address=45.91.116.5:4320 , piece_id=IKUCNDOTLMU4RDFQW5BHULQQBATM4UYNR7E5RXPFDUZQ53OXV37Q, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399737294 (203h27m11.358834533s) [noise=true , address=104.36.82.4:4818 , piece_id=QWKWGTQASLFES4NHZDQAMZYGSMU2T5MC7LHWVT7YFBKX4JAOKWBQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399753936 (203h27m11.358851175s) [noise=true , address=45.91.116.14:4203 , piece_id=ZJEVJ4DRTQQSAU2NUTEBPXAMUS3RAZ34D7B2ZPAL26D22XEPKYAQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.401333446 (203h27m11.360430683s) [noise=true , address=77.90.31.11:4562 , piece_id=7UCJ2NOZELQIZAI4VTWTHAZYNELMWKV2PDERTVY4YGG4KKOWXWHQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.404921853 (203h27m11.364019102s) [noise=true , address=104.36.82.40:4962 , piece_id=6KUC3GJQ4MCNQW4665FNPUSR3SSKP7XKSO2M63XZJW7UBKCW5YPA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.405663484 (203h27m11.364760721s) [noise=true , address=104.36.82.16:4308 , piece_id=XTNKSQXKFAYLKO5ASHMS2XWHR2ICMTGHMZTJGI47UUBA4DUYTWSA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.409136020 (203h27m11.36823327s) [noise=true , address=45.91.116.10:4098 , piece_id=7ZFFIBNOJ4GLKSZPKAUGKCVC2I7W5C3FQAXYYWXIIL7O7OARVBKQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.425800042 (203h27m11.384897277s) [noise=true , address=104.36.82.21:4508 , piece_id=AJ2ZSCAXMJB7HMHTSXUDQZJBZPCLGEBYUC2IEGTEGTB2BGHNCJCQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.426532877 (203h27m11.385630109s) [noise=true , address=104.36.82.59:4325 , piece_id=VUO4P5OLQRTK3BIC5TRA37TKAFV6KOMA25PVDIWHBKV7NO32ZMPQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.430052219 (203h27m11.389149451s) [noise=true , address=77.90.31.18:4824 , piece_id=OAZDEBEEPYEAKQWPF7KGP4YTR5PVOHD4XO5RSLPJ653IXFJTVCDA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.441103649 (203h27m11.400200886s) [noise=true , address=104.36.82.23:4472 , piece_id=7BR3ZF7MIB5K3GE5OBUHWHK7QTE3LJIHCN5B7DZLPCD5G7FNNYRA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.465085358 (203h27m11.424182604s) [noise=true , address=77.90.31.21:4884 , piece_id=43HVGGMFGGUSWZ3P6YRZXRHOHGUM2JS727HMHKNZ4V5RQYZSUVRQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.538072946 (203h27m11.497170192s) [noise=true , address=104.36.82.60:4342 , piece_id=A36WWC2UFQKGYZHKOTNVRF7EHESDRIH7V2LLKLVLLNPTUAXLNQ5A, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.538136336 (203h27m11.497233584s) [seg_pos={0 0}, upload=211774]: Segment reached optimal threshold of 14 pieces. Aug 17 04:52:49.538350773 (203h27m11.49744801s) [noise=true , address=104.36.82.44:4904 , piece_id=6S5LDTNJP226KVAB2I6PNFZDKHKH4IQD5MRRBBBZVQKQYWUDPNEQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538527792 (203h27m11.497625032s) [noise=true , address=79.127.201.215:4360 , piece_id=AHN3MJ6MKAKQ7ALTI5Z642BCZLITRLRPY4IIA2N3DE5RMP3RTGGQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y7dgLXYLTGGFXzHQ5ZYJ4f4kWYCoyiBhGesd1xqbTbnYjxNzG): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538557176 (203h27m11.497654422s) [noise=true , address=77.90.31.19:4858 , piece_id=3IFWCPGWHNPOWK32CGAYVH7DRRKL3KY57UBJ4RJNJEGKXS34OBKA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538625497 (203h27m11.497722733s) [noise=true , address=104.36.82.5:4866 , piece_id=SK4UJRQUOQZQVQI77ZBUPZ6BOKI3WNDPZHR5QDUF34OOKENI5EGQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection; write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection Aug 17 04:52:49.538649790 (203h27m11.497747036s) [seg_pos={0 0}, upload=211774]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:52:49.538727663 (203h27m11.497824908s) [upload=211774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:52:49.538732662 (203h27m11.497829907s) [upload=211774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:49.694817003 (203h27m11.653914249s) [upload=211774]: Done uploading segments... Aug 17 04:52:49.694852660 (203h27m11.653949907s) [upload=211774]: Upload finished. err: <nil> Duration: 2.469025209s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.X........X...............] duration: 820.757387ms FAIL (2025-08-17 05:02:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:52:29.864918009 (203h26m51.824015266s) [upload=211769]: Starting upload Aug 17 04:52:29.864987814 (203h26m51.824085071s) [upload=211769]: Done starting upload Aug 17 04:52:29.865067633 (203h26m51.82416488s) [upload=211769]: Uploading segments... Aug 17 04:52:29.865154190 (203h26m51.824251449s) [upload=211769]: Got next segment. Inline: true Aug 17 04:52:29.865161885 (203h26m51.824259137s) [upload=211769]: Waiting for error group managing segments... Aug 17 04:52:29.865211769 (203h26m51.824309016s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:29.865218162 (203h26m51.824315412s) [upload=211769]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 17 04:52:29.865222666 (203h26m51.824319916s) [upload=211769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:29.961406445 (203h26m51.920503696s) [upload=211769]: Done uploading segments... Aug 17 04:52:29.961431277 (203h26m51.920528524s) [upload=211769]: Upload finished. err: metaclient: Too Many Requests Duration:* 820.789738ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 3.014337448s FAIL (2025-08-17 05:02:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.014429627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XXXXX...............................................] duration: 2.799464035s FAIL (2025-08-17 05:02:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.799490345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.........................X............] duration: 3.46424522s FAIL (2025-08-17 05:02:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== Aug 17 04:52:48.490540025 (203h27m10.449637277s) [upload=211774]: Starting upload Aug 17 04:52:48.490604066 (203h27m10.449701325s) [upload=211774]: Done starting upload Aug 17 04:52:48.490733030 (203h27m10.449830268s) [upload=211774]: Uploading segments... Aug 17 04:52:48.490895113 (203h27m10.449992358s) [upload=211774]: Got next segment. Inline: false Aug 17 04:52:48.490944376 (203h27m10.450041721s) [upload=211774]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:52:48.490952125 (203h27m10.450049373s) [upload=211774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:52:48.694225741 (203h27m10.653322987s) [seg_pos={0 0}, upload=211774]: Begin upload segment... Aug 17 04:52:48.694287146 (203h27m10.653384386s) [seg_pos={0 0}, upload=211774]: Done begin upload segment. Aug 17 04:52:48.694324474 (203h27m10.653421723s) [noise=true , address=77.90.31.11:4562 , piece_id=7UCJ2NOZELQIZAI4VTWTHAZYNELMWKV2PDERTVY4YGG4KKOWXWHQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694369283 (203h27m10.653466547s) [noise=true , address=77.90.31.18:4824 , piece_id=OAZDEBEEPYEAKQWPF7KGP4YTR5PVOHD4XO5RSLPJ653IXFJTVCDA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694778226 (203h27m10.653875472s) [noise=true , address=79.127.201.215:4360 , piece_id=AHN3MJ6MKAKQ7ALTI5Z642BCZLITRLRPY4IIA2N3DE5RMP3RTGGQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.694866587 (203h27m10.653963831s) [noise=true , address=104.36.82.39:4738 , piece_id=ZIJ4NHPJY5WWE46LEZDUCULNAAWR363RJUT2DBM25RC4YEHY47YQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695133229 (203h27m10.654230476s) [noise=true , address=104.36.82.21:4508 , piece_id=AJ2ZSCAXMJB7HMHTSXUDQZJBZPCLGEBYUC2IEGTEGTB2BGHNCJCQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695323197 (203h27m10.654420443s) [noise=true , address=104.36.82.5:4866 , piece_id=SK4UJRQUOQZQVQI77ZBUPZ6BOKI3WNDPZHR5QDUF34OOKENI5EGQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695511156 (203h27m10.654608403s) [noise=true , address=104.36.82.23:4472 , piece_id=7BR3ZF7MIB5K3GE5OBUHWHK7QTE3LJIHCN5B7DZLPCD5G7FNNYRA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695658635 (203h27m10.654755879s) [noise=true , address=104.36.82.4:4818 , piece_id=QWKWGTQASLFES4NHZDQAMZYGSMU2T5MC7LHWVT7YFBKX4JAOKWBQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.695902387 (203h27m10.654999635s) [noise=true , address=45.91.116.10:4098 , piece_id=7ZFFIBNOJ4GLKSZPKAUGKCVC2I7W5C3FQAXYYWXIIL7O7OARVBKQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696122792 (203h27m10.655220037s) [noise=true , address=77.90.31.21:4884 , piece_id=43HVGGMFGGUSWZ3P6YRZXRHOHGUM2JS727HMHKNZ4V5RQYZSUVRQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696279899 (203h27m10.655377147s) [noise=true , address=104.36.82.44:4904 , piece_id=6S5LDTNJP226KVAB2I6PNFZDKHKH4IQD5MRRBBBZVQKQYWUDPNEQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696603169 (203h27m10.655700414s) [noise=true , address=104.36.82.59:4325 , piece_id=VUO4P5OLQRTK3BIC5TRA37TKAFV6KOMA25PVDIWHBKV7NO32ZMPQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.696649532 (203h27m10.65574678s) [noise=true , address=104.36.82.40:4962 , piece_id=6KUC3GJQ4MCNQW4665FNPUSR3SSKP7XKSO2M63XZJW7UBKCW5YPA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697180139 (203h27m10.656277387s) [noise=true , address=45.91.116.14:4203 , piece_id=ZJEVJ4DRTQQSAU2NUTEBPXAMUS3RAZ34D7B2ZPAL26D22XEPKYAQ, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697194676 (203h27m10.65629196s) [noise=true , address=77.90.31.19:4858 , piece_id=3IFWCPGWHNPOWK32CGAYVH7DRRKL3KY57UBJ4RJNJEGKXS34OBKA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697581706 (203h27m10.656678954s) [noise=true , address=104.36.82.60:4342 , piece_id=A36WWC2UFQKGYZHKOTNVRF7EHESDRIH7V2LLKLVLLNPTUAXLNQ5A, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697725008 (203h27m10.656822252s) [noise=true , address=45.91.116.5:4320 , piece_id=IKUCNDOTLMU4RDFQW5BHULQQBATM4UYNR7E5RXPFDUZQ53OXV37Q, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:48.697903656 (203h27m10.657000947s) [noise=true , address=104.36.82.16:4308 , piece_id=XTNKSQXKFAYLKO5ASHMS2XWHR2ICMTGHMZTJGI47UUBA4DUYTWSA, seg_pos={0 0}, upload=211774]: Uploading piece... Aug 17 04:52:49.240699808 (203h27m11.199797041s) [upload=211774]: Next returned nil segment Aug 17 04:52:49.240756486 (203h27m11.199853836s) [upload=211774]: Waiting for error group managing segments... Aug 17 04:52:49.399120274 (203h27m11.35821751s) [noise=true , address=104.36.82.39:4738 , piece_id=ZIJ4NHPJY5WWE46LEZDUCULNAAWR363RJUT2DBM25RC4YEHY47YQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399144035 (203h27m11.358241275s) [noise=true , address=45.91.116.5:4320 , piece_id=IKUCNDOTLMU4RDFQW5BHULQQBATM4UYNR7E5RXPFDUZQ53OXV37Q, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399737294 (203h27m11.358834533s) [noise=true , address=104.36.82.4:4818 , piece_id=QWKWGTQASLFES4NHZDQAMZYGSMU2T5MC7LHWVT7YFBKX4JAOKWBQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.399753936 (203h27m11.358851175s) [noise=true , address=45.91.116.14:4203 , piece_id=ZJEVJ4DRTQQSAU2NUTEBPXAMUS3RAZ34D7B2ZPAL26D22XEPKYAQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.401333446 (203h27m11.360430683s) [noise=true , address=77.90.31.11:4562 , piece_id=7UCJ2NOZELQIZAI4VTWTHAZYNELMWKV2PDERTVY4YGG4KKOWXWHQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.404921853 (203h27m11.364019102s) [noise=true , address=104.36.82.40:4962 , piece_id=6KUC3GJQ4MCNQW4665FNPUSR3SSKP7XKSO2M63XZJW7UBKCW5YPA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.405663484 (203h27m11.364760721s) [noise=true , address=104.36.82.16:4308 , piece_id=XTNKSQXKFAYLKO5ASHMS2XWHR2ICMTGHMZTJGI47UUBA4DUYTWSA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.409136020 (203h27m11.36823327s) [noise=true , address=45.91.116.10:4098 , piece_id=7ZFFIBNOJ4GLKSZPKAUGKCVC2I7W5C3FQAXYYWXIIL7O7OARVBKQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.425800042 (203h27m11.384897277s) [noise=true , address=104.36.82.21:4508 , piece_id=AJ2ZSCAXMJB7HMHTSXUDQZJBZPCLGEBYUC2IEGTEGTB2BGHNCJCQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.426532877 (203h27m11.385630109s) [noise=true , address=104.36.82.59:4325 , piece_id=VUO4P5OLQRTK3BIC5TRA37TKAFV6KOMA25PVDIWHBKV7NO32ZMPQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.430052219 (203h27m11.389149451s) [noise=true , address=77.90.31.18:4824 , piece_id=OAZDEBEEPYEAKQWPF7KGP4YTR5PVOHD4XO5RSLPJ653IXFJTVCDA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.441103649 (203h27m11.400200886s) [noise=true , address=104.36.82.23:4472 , piece_id=7BR3ZF7MIB5K3GE5OBUHWHK7QTE3LJIHCN5B7DZLPCD5G7FNNYRA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.465085358 (203h27m11.424182604s) [noise=true , address=77.90.31.21:4884 , piece_id=43HVGGMFGGUSWZ3P6YRZXRHOHGUM2JS727HMHKNZ4V5RQYZSUVRQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.538072946 (203h27m11.497170192s) [noise=true , address=104.36.82.60:4342 , piece_id=A36WWC2UFQKGYZHKOTNVRF7EHESDRIH7V2LLKLVLLNPTUAXLNQ5A, seg_pos={0 0}, upload=211774]: Done uploading piece. err: <nil> Aug 17 04:52:49.538136336 (203h27m11.497233584s) [seg_pos={0 0}, upload=211774]: Segment reached optimal threshold of 14 pieces. Aug 17 04:52:49.538350773 (203h27m11.49744801s) [noise=true , address=104.36.82.44:4904 , piece_id=6S5LDTNJP226KVAB2I6PNFZDKHKH4IQD5MRRBBBZVQKQYWUDPNEQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538527792 (203h27m11.497625032s) [noise=true , address=79.127.201.215:4360 , piece_id=AHN3MJ6MKAKQ7ALTI5Z642BCZLITRLRPY4IIA2N3DE5RMP3RTGGQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y7dgLXYLTGGFXzHQ5ZYJ4f4kWYCoyiBhGesd1xqbTbnYjxNzG): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538557176 (203h27m11.497654422s) [noise=true , address=77.90.31.19:4858 , piece_id=3IFWCPGWHNPOWK32CGAYVH7DRRKL3KY57UBJ4RJNJEGKXS34OBKA, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c): protocol: expected piece hash; context canceled; EOF Aug 17 04:52:49.538625497 (203h27m11.497722733s) [noise=true , address=104.36.82.5:4866 , piece_id=SK4UJRQUOQZQVQI77ZBUPZ6BOKI3WNDPZHR5QDUF34OOKENI5EGQ, seg_pos={0 0}, upload=211774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection; write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:34462->104.36.82.5:4866: use of closed network connection Aug 17 04:52:49.538649790 (203h27m11.497747036s) [seg_pos={0 0}, upload=211774]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 17 04:52:49.538727663 (203h27m11.497824908s) [upload=211774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 17 04:52:49.538732662 (203h27m11.497829907s) [upload=211774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 17 04:52:49.694817003 (203h27m11.653914249s) [upload=211774]: Done uploading segments... Aug 17 04:52:49.694852660 (203h27m11.653949907s) [upload=211774]: Upload finished. err: <nil> Duration: 3.464275153s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 2.718431006s FAIL (2025-08-17 05:02:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.718459205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 2.770874339s FAIL (2025-08-17 05:02:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.770907173s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)