Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 354467 reports (Page 576 of 7090)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..........................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXX] duration: 5.000806844s trace: https://jaeger.dev.storj.io/trace/49259f8c68d48702 FAIL (2026-03-18 19:58:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000829009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..............................XXX.XXX...............X.......] duration: 17.500349928s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:58:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.500377676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...............................XXX.XXX...............X......] duration: 17.084774861s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:58:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.084799791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................X..X...X..X..X........X..X..X.] duration: 16.668426703s trace: https://jaeger.dev.storj.io/trace/657fdcaa051c01a8 FAIL (2026-03-18 19:58:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18 19:56:59.292408464 (195h46m2.308078058s) [upload=188049]: Uploading segments... Mar 18 19:56:59.292755979 (195h46m2.30842558s) [upload=188049]: Got next segment. Inline: false Mar 18 19:56:59.292808462 (195h46m2.308478061s) [upload=188049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:56:59.292813871 (195h46m2.308483471s) [upload=188049]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:57:02.876720251 (195h46m5.892389863s) [seg_pos={0 0}, upload=188049]: Begin upload segment... Mar 18 19:57:02.876793971 (195h46m5.892463581s) [seg_pos={0 0}, upload=188049]: Done begin upload segment. Mar 18 19:57:02.876833263 (195h46m5.892502873s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.876890974 (195h46m5.892560585s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.877956175 (195h46m5.893625785s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.878298073 (195h46m5.89396768s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.879712704 (195h46m5.895382315s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.880345930 (195h46m5.896015539s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.880748175 (195h46m5.896417775s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.881290748 (195h46m5.896960358s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.882022155 (195h46m5.897691756s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.882473710 (195h46m5.898143318s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.883003620 (195h46m5.898673224s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.883504542 (195h46m5.899174149s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884006474 (195h46m5.899676078s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884343043 (195h46m5.900012643s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884660141 (195h46m5.900329745s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.885052951 (195h46m5.900722557s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.886175686 (195h46m5.901845308s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.886594862 (195h46m5.902264463s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:03.444101668 (195h46m6.459771278s) [upload=188049]: Next returned nil segment Mar 18 19:57:03.444124472 (195h46m6.459794082s) [upload=188049]: Waiting for error group managing segments... Mar 18 19:57:03.590092875 (195h46m6.605762476s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.603331674 (195h46m6.619001274s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.613170926 (195h46m6.628840529s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.617370351 (195h46m6.633039952s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.687019561 (195h46m6.702689162s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.967066275 (195h46m6.982735877s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.977678776 (195h46m6.993348378s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.986006395 (195h46m7.001676s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.105385143 (195h46m7.121054743s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.113300230 (195h46m7.12896983s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.156743026 (195h46m7.172412637s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.212855060 (195h46m7.22852466s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.223915835 (195h46m7.239585447s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:59.001554644 (195h47m2.017224275s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.001715114 (195h47m2.017384752s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1X6WKD6kuW1CgatniwYHDWrT9tfdq9sxB25UxAUBzTRXsJPyfH, address:145.79.227.32:4846): protocol: rpc: dial tcp 145.79.227.32:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002131998 (195h47m2.017801609s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002164582 (195h47m2.017834216s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:145.79.227.36:4538): protocol: rpc: dial tcp 145.79.227.36:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002428976 (195h47m2.018098586s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:145.79.227.28:4749): protocol: rpc: dial tcp 145.79.227.28:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002458729 (195h47m2.01812833s) [seg_pos={0 0}, upload=188049]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:57:59.118230032 (195h47m2.133899643s) [upload=188049]: Done uploading segments... Mar 18 19:57:59.118301627 (195h47m2.133971217s) [upload=188049]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.668453435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............................X..X...X..X..X........X..X..X] duration: 1m0.119023148s trace: https://jaeger.dev.storj.io/trace/657fdcaa051c01a8 FAIL (2026-03-18 19:57:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18 19:56:59.292408464 (195h46m2.308078058s) [upload=188049]: Uploading segments... Mar 18 19:56:59.292755979 (195h46m2.30842558s) [upload=188049]: Got next segment. Inline: false Mar 18 19:56:59.292808462 (195h46m2.308478061s) [upload=188049]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:56:59.292813871 (195h46m2.308483471s) [upload=188049]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:57:02.876720251 (195h46m5.892389863s) [seg_pos={0 0}, upload=188049]: Begin upload segment... Mar 18 19:57:02.876793971 (195h46m5.892463581s) [seg_pos={0 0}, upload=188049]: Done begin upload segment. Mar 18 19:57:02.876833263 (195h46m5.892502873s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.876890974 (195h46m5.892560585s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.877956175 (195h46m5.893625785s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.878298073 (195h46m5.89396768s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.879712704 (195h46m5.895382315s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.880345930 (195h46m5.896015539s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.880748175 (195h46m5.896417775s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.881290748 (195h46m5.896960358s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.882022155 (195h46m5.897691756s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.882473710 (195h46m5.898143318s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.883003620 (195h46m5.898673224s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.883504542 (195h46m5.899174149s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884006474 (195h46m5.899676078s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884343043 (195h46m5.900012643s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.884660141 (195h46m5.900329745s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.885052951 (195h46m5.900722557s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.886175686 (195h46m5.901845308s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:02.886594862 (195h46m5.902264463s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Uploading piece... Mar 18 19:57:03.444101668 (195h46m6.459771278s) [upload=188049]: Next returned nil segment Mar 18 19:57:03.444124472 (195h46m6.459794082s) [upload=188049]: Waiting for error group managing segments... Mar 18 19:57:03.590092875 (195h46m6.605762476s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.603331674 (195h46m6.619001274s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.613170926 (195h46m6.628840529s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.617370351 (195h46m6.633039952s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.687019561 (195h46m6.702689162s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.967066275 (195h46m6.982735877s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.977678776 (195h46m6.993348378s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:03.986006395 (195h46m7.001676s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.105385143 (195h46m7.121054743s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.113300230 (195h46m7.12896983s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.156743026 (195h46m7.172412637s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.212855060 (195h46m7.22852466s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:04.223915835 (195h46m7.239585447s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil> Mar 18 19:57:59.001554644 (195h47m2.017224275s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.001715114 (195h47m2.017384752s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1X6WKD6kuW1CgatniwYHDWrT9tfdq9sxB25UxAUBzTRXsJPyfH, address:145.79.227.32:4846): protocol: rpc: dial tcp 145.79.227.32:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002131998 (195h47m2.017801609s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002164582 (195h47m2.017834216s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:145.79.227.36:4538): protocol: rpc: dial tcp 145.79.227.36:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002428976 (195h47m2.018098586s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:145.79.227.28:4749): protocol: rpc: dial tcp 145.79.227.28:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:59.002458729 (195h47m2.01812833s) [seg_pos={0 0}, upload=188049]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:57:59.118230032 (195h47m2.133899643s) [upload=188049]: Done uploading segments... Mar 18 19:57:59.118301627 (195h47m2.133971217s) [upload=188049]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.119048585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX] duration: 30.11317306s trace: https://jaeger.dev.storj.io/trace/64ac3f309d0ac30b FAIL (2026-03-18 19:57:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== Mar 18 19:57:27.576180116 (195h46m30.591849717s) [upload=188056]: Starting upload Mar 18 19:57:27.576260950 (195h46m30.59193055s) [upload=188056]: Done starting upload Mar 18 19:57:27.576308583 (195h46m30.591978183s) [upload=188056]: Uploading segments... Mar 18 19:57:27.576638834 (195h46m30.592308437s) [upload=188056]: Got next segment. Inline: false Mar 18 19:57:27.576767837 (195h46m30.592437437s) [upload=188056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:57:27.576775316 (195h46m30.592444906s) [upload=188056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:57:30.526375968 (195h46m33.542045568s) [seg_pos={0 0}, upload=188056]: Begin upload segment... Mar 18 19:57:30.526441519 (195h46m33.542111119s) [seg_pos={0 0}, upload=188056]: Done begin upload segment. Mar 18 19:57:30.526468577 (195h46m33.542138181s) [noise=true , address=77.90.31.18:4808 , piece_id=PXLKLYFSZKTM52F65NBXBXBQIVSOJ5N4P5L2A5POELVEW63BZGYA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.529277868 (195h46m33.544947477s) [noise=true , address=79.127.219.41:4925 , piece_id=NXEDLNXOEGX3YMJDEMXGARDEO2JSIJ474IKPJKDI5A6XAHZMZNWA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.530452299 (195h46m33.546121903s) [noise=true , address=145.79.227.21:4474 , piece_id=LPU7ACTAJR6BRISTGS4LOJVOD7YGCUSRGAYXQO2DQ7JTG7I5RZ2Q, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.532462690 (195h46m33.548132292s) [noise=true , address=145.79.227.40:4053 , piece_id=RV2EWYVZNQ33TQNRILNTCN3M6A3E6U7GTLKH427XFR3UDD4KPPFA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.535953831 (195h46m33.551623431s) [noise=true , address=45.91.116.9:4264 , piece_id=X7JOYMNH6ZZB6KYCTGZAT6TAIHCA4I5UIULTLAPYSWKGYMKJD5RA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.536561835 (195h46m33.552231443s) [noise=true , address=145.79.227.30:4762 , piece_id=SGUKVW3H6C756J7XAF5WDOLMVCYXLMEOWHK65PLBMJ2ZVPJWW3TA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.538576676 (195h46m33.554246282s) [noise=true , address=145.79.227.37:4566 , piece_id=OX7TKCMMB4OZOYP4T6XWAAKTM4XPJFZF47MGCXF7SBRUOMMXA3WA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.539028198 (195h46m33.554697799s) [noise=true , address=77.90.31.14:4661 , piece_id=QXHNAULMBYIWOHY2EUMU4YKJ5E7IL4I66SM7HZAB2YUWFATIB4UA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.540341808 (195h46m33.556011412s) [noise=true , address=45.91.116.4:4955 , piece_id=A3E7IAXX3V26I6OCNYCQR7EABIIGWXLGB2G2ZLZBH757XTNWGP3Q, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.541390056 (195h46m33.557059666s) [noise=true , address=145.79.227.44:4193 , piece_id=45DIDLPGRA4YCTOP2AQFR4YWXAG4UQONNC5Y5T35J3KAFGEXV4XQ, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.542025467 (195h46m33.557695075s) [noise=true , address=45.91.116.5:4300 , piece_id=7PZMFCBTBSOZ22J3UY4YVB7R7VCEWW2HGZYC7Z6JTX2HXRKDFQGQ, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.543977728 (195h46m33.559647334s) [noise=true , address=77.90.31.10:4517 , piece_id=AXQXWOXDUO4CG256S2KYJOQYMFPLHMFOMFYNFLJKX33O6EIGR67Q, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.546456539 (195h46m33.562126149s) [noise=true , address=77.90.31.15:4701 , piece_id=W2MFHCCHB6J6DJVJSPFYFMTXKM6NZIVNTP3OO7SYYR6RV5DNBPLQ, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.546641079 (195h46m33.562310686s) [noise=true , address=145.79.227.14:4841 , piece_id=OJ5FQDIZJWQSLGOIQYOQEFNSPJASCPHBYXO5P3Q7B675AD5UGTQQ, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.549498288 (195h46m33.565167891s) [noise=true , address=145.79.227.49:4338 , piece_id=MWK5GSASKFLYHRPGA7YIXTIZUEEUFRJBPQQVX3VFJKDNY6BB77JA, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:30.549743281 (195h46m33.56541289s) [noise=true , address=145.79.227.33:4869 , piece_id=NFLLSAOZBPNJC64O7V33MU5IP5EQNWHW3F3AXKDEQR2QML6IGROQ, seg_pos={0 0}, upload=188056]: Uploading piece... Mar 18 19:57:31.157514106 (195h46m34.173183707s) [upload=188056]: Next returned nil segment Mar 18 19:57:31.157534741 (195h46m34.173204341s) [upload=188056]: Waiting for error group managing segments... Mar 18 19:57:31.284819743 (195h46m34.30048935s) [noise=true , address=45.91.116.9:4264 , piece_id=X7JOYMNH6ZZB6KYCTGZAT6TAIHCA4I5UIULTLAPYSWKGYMKJD5RA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.306900617 (195h46m34.322570212s) [noise=true , address=45.91.116.4:4955 , piece_id=A3E7IAXX3V26I6OCNYCQR7EABIIGWXLGB2G2ZLZBH757XTNWGP3Q, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.311955594 (195h46m34.327625197s) [noise=true , address=79.127.219.41:4925 , piece_id=NXEDLNXOEGX3YMJDEMXGARDEO2JSIJ474IKPJKDI5A6XAHZMZNWA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.312719438 (195h46m34.328389041s) [noise=true , address=45.91.116.5:4300 , piece_id=7PZMFCBTBSOZ22J3UY4YVB7R7VCEWW2HGZYC7Z6JTX2HXRKDFQGQ, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.314182529 (195h46m34.329852131s) [noise=true , address=77.90.31.10:4517 , piece_id=AXQXWOXDUO4CG256S2KYJOQYMFPLHMFOMFYNFLJKX33O6EIGR67Q, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.345812426 (195h46m34.36148203s) [noise=true , address=77.90.31.18:4808 , piece_id=PXLKLYFSZKTM52F65NBXBXBQIVSOJ5N4P5L2A5POELVEW63BZGYA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.363225128 (195h46m34.378894728s) [noise=true , address=77.90.31.14:4661 , piece_id=QXHNAULMBYIWOHY2EUMU4YKJ5E7IL4I66SM7HZAB2YUWFATIB4UA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:31.388192524 (195h46m34.403862124s) [noise=true , address=77.90.31.15:4701 , piece_id=W2MFHCCHB6J6DJVJSPFYFMTXKM6NZIVNTP3OO7SYYR6RV5DNBPLQ, seg_pos={0 0}, upload=188056]: Done uploading piece. err: <nil> Mar 18 19:57:57.280118142 (195h47m0.295787774s) [noise=true , address=145.79.227.49:4338 , piece_id=MWK5GSASKFLYHRPGA7YIXTIZUEEUFRJBPQQVX3VFJKDNY6BB77JA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.280219072 (195h47m0.295888681s) [noise=true , address=145.79.227.37:4566 , piece_id=OX7TKCMMB4OZOYP4T6XWAAKTM4XPJFZF47MGCXF7SBRUOMMXA3WA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, address:145.79.227.37:4566): protocol: rpc: dial tcp 145.79.227.37:4566: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.280460697 (195h47m0.296130307s) [noise=true , address=145.79.227.14:4841 , piece_id=OJ5FQDIZJWQSLGOIQYOQEFNSPJASCPHBYXO5P3Q7B675AD5UGTQQ, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:12ndNj9Dy8qQSbHXwVWw31sz19ki1qy6Zgetjvj1A5Q3odkzmDm, address:145.79.227.14:4841): protocol: rpc: dial tcp 145.79.227.14:4841: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.280786748 (195h47m0.296456353s) [noise=true , address=145.79.227.21:4474 , piece_id=LPU7ACTAJR6BRISTGS4LOJVOD7YGCUSRGAYXQO2DQ7JTG7I5RZ2Q, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: rpc: dial tcp 145.79.227.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.281139911 (195h47m0.29680952s) [noise=true , address=145.79.227.40:4053 , piece_id=RV2EWYVZNQ33TQNRILNTCN3M6A3E6U7GTLKH427XFR3UDD4KPPFA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:145.79.227.40:4053): protocol: rpc: dial tcp 145.79.227.40:4053: i/o timeout; conn closed: op Mar 18 19:57:57.281311912 (195h47m0.296981515s) [noise=true , address=145.79.227.44:4193 , piece_id=45DIDLPGRA4YCTOP2AQFR4YWXAG4UQONNC5Y5T35J3KAFGEXV4XQ, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: rpc: dial tcp 145.79.227.44:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.281382560 (195h47m0.297052169s) [noise=true , address=145.79.227.30:4762 , piece_id=SGUKVW3H6C756J7XAF5WDOLMVCYXLMEOWHK65PLBMJ2ZVPJWW3TA, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: rpc: dial tcp 145.79.227.30:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.281560552 (195h47m0.297230155s) [noise=true , address=145.79.227.33:4869 , piece_id=NFLLSAOZBPNJC64O7V33MU5IP5EQNWHW3F3AXKDEQR2QML6IGROQ, seg_pos={0 0}, upload=188056]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: rpc: dial tcp 145.79.227.33:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:57:57.281697501 (195h47m0.297367102s) [seg_pos={0 0}, upload=188056]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Mar 18 19:57:57.390707284 (195h47m0.406376891s) [upload=188056]: Done uploading segments... Mar 18 19:57:57.390771821 (195h47m0.406441433s) [upload=188056]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.113196629s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [................................XXX.XXX...............X.....] duration: 18.321240604s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:57:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.321281379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [...........................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXX] duration: 5.001678493s trace: https://jaeger.dev.storj.io/trace/1bc36ebef9356c7d FAIL (2026-03-18 19:57:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001707005s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.................................XXX.XXX...............X....] duration: 18.33852173s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:57:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.338549905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [............................XXXXXXXX....XX.XXXXXXXXXXXXXXXXX] duration: 5.000858577s trace: https://jaeger.dev.storj.io/trace/4310948e63c0c25e FAIL (2026-03-18 19:56:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000882214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXX] duration: 30.105886295s trace: https://jaeger.dev.storj.io/trace/5670d7c71354b4 FAIL (2026-03-18 19:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 19:56:14.445981196 (195h45m17.4616508s) [upload=188041]: Starting upload Mar 18 19:56:14.446061390 (195h45m17.461730995s) [upload=188041]: Done starting upload Mar 18 19:56:14.446309733 (195h45m17.461979336s) [upload=188041]: Uploading segments... Mar 18 19:56:14.446629520 (195h45m17.46229912s) [upload=188041]: Got next segment. Inline: false Mar 18 19:56:14.446683030 (195h45m17.462352644s) [upload=188041]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:56:14.446689456 (195h45m17.46235906s) [upload=188041]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:56:17.769208821 (195h45m20.784878427s) [seg_pos={0 0}, upload=188041]: Begin upload segment... Mar 18 19:56:17.769277419 (195h45m20.784947019s) [seg_pos={0 0}, upload=188041]: Done begin upload segment. Mar 18 19:56:17.769314340 (195h45m20.78498395s) [noise=true , address=145.79.227.33:4874 , piece_id=TV6PQHDDRE4UNGWQDKTCAZERHO6LSN433LWIG4X244P3VKDEADYQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.769368319 (195h45m20.785037928s) [noise=true , address=77.90.31.11:4558 , piece_id=A6I5XSKU7LWBZIJMD3NVC3YHA5663QIARPJX4HZAQL7WGL24QOJA, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.770065733 (195h45m20.785735342s) [noise=true , address=77.90.31.16:4737 , piece_id=CHS4E5YFJ4E6773WCFLBRUFEISWY75EVQ46SUB7KIWZSDPJ3L4XA, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.770334609 (195h45m20.786004216s) [noise=true , address=77.90.31.14:4663 , piece_id=ZHRSWG5OJZHAMHWIX5BJLHSFE5SDJEFRDWGFXFIQGPZWR3SLVJUA, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.770784597 (195h45m20.786454207s) [noise=true , address=79.127.219.38:4820 , piece_id=5OBEYN65AKQQJXKRUCQTMB7WVQ7Z3ZPEJIBMZHWDHHITGLKQJJ5A, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.771269513 (195h45m20.78693912s) [noise=true , address=145.79.227.3:4155 , piece_id=4L42QGI32QXDUAY3RCE5TQUQZYYG4SSQB2GGSPJAJH6MXAMJVSAQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.771757508 (195h45m20.787427117s) [noise=true , address=45.91.116.9:4265 , piece_id=RSMSUTCDORXWE223BWD4YNP7KMV6H2K2HZIXL457QYXHUNEDASTQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.772558287 (195h45m20.788227887s) [noise=true , address=45.91.116.10:4081 , piece_id=QKUEKRSUKSRK5QUAKUIAYZKE3MVVTMJSOOI4UBRQ3OEVITPPUZ4A, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.772686440 (195h45m20.788356041s) [noise=true , address=79.127.219.45:4856 , piece_id=6YNPHXN42DTWSMURYVLWUHVNFYWLQWAWWKEEDTWQ43UNV7PXSBEQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.773862418 (195h45m20.789532026s) [noise=true , address=145.79.227.44:4198 , piece_id=WX34JQXE2C2G3WCBO365G7NHNMA5AVXNL5LBL6LJSNNZRYLZPQSQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.774133820 (195h45m20.789803423s) [noise=true , address=45.91.116.4:4947 , piece_id=GRBU5GIKTVOQBPQGJILQJSIAKX5MENU3WXQSU5SZPZDQC5ZSX7XQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.775160309 (195h45m20.790829918s) [noise=true , address=77.90.31.18:4805 , piece_id=AYKQ2ZSEGER7UH5B7VNO34C5GQOTSJBO7SUTSV6PTGBQ5ZKTGWGQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.775405321 (195h45m20.791074932s) [noise=true , address=77.90.31.10:4522 , piece_id=UTCSGAQB7PGA7XYJPZDWPXFEE2CDKKR4TIR5RIRWUNO7AIZ4R2CQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.775909205 (195h45m20.791578812s) [noise=true , address=45.91.116.5:4298 , piece_id=BGSEHZYYOCOCWYZFIRTLJAQTNIU5VFOZLT4T55JY2YKAAFQJLZPQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.777767319 (195h45m20.793436919s) [noise=true , address=45.91.116.11:4119 , piece_id=UOMKJYXLGJQR5IV23Z7XQ5BVVHU5IBODONKK4TE6UVSBR4MBN5LA, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:17.778186092 (195h45m20.793855692s) [noise=true , address=145.79.227.52:4448 , piece_id=6VQISZH4PYG7T67VBVQALI2K5FYHOBA63UVQWNZ752OHVS4V7KEQ, seg_pos={0 0}, upload=188041]: Uploading piece... Mar 18 19:56:18.372155106 (195h45m21.387824697s) [upload=188041]: Next returned nil segment Mar 18 19:56:18.372170249 (195h45m21.387839852s) [upload=188041]: Waiting for error group managing segments... Mar 18 19:56:18.498843913 (195h45m21.514513512s) [noise=true , address=77.90.31.11:4558 , piece_id=A6I5XSKU7LWBZIJMD3NVC3YHA5663QIARPJX4HZAQL7WGL24QOJA, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.503179302 (195h45m21.518848905s) [noise=true , address=77.90.31.10:4522 , piece_id=UTCSGAQB7PGA7XYJPZDWPXFEE2CDKKR4TIR5RIRWUNO7AIZ4R2CQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.508961781 (195h45m21.524631382s) [noise=true , address=45.91.116.5:4298 , piece_id=BGSEHZYYOCOCWYZFIRTLJAQTNIU5VFOZLT4T55JY2YKAAFQJLZPQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.516694479 (195h45m21.53236407s) [noise=true , address=45.91.116.9:4265 , piece_id=RSMSUTCDORXWE223BWD4YNP7KMV6H2K2HZIXL457QYXHUNEDASTQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.520132940 (195h45m21.53580254s) [noise=true , address=79.127.219.38:4820 , piece_id=5OBEYN65AKQQJXKRUCQTMB7WVQ7Z3ZPEJIBMZHWDHHITGLKQJJ5A, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.522471652 (195h45m21.538141253s) [noise=true , address=45.91.116.4:4947 , piece_id=GRBU5GIKTVOQBPQGJILQJSIAKX5MENU3WXQSU5SZPZDQC5ZSX7XQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.524528754 (195h45m21.540198358s) [noise=true , address=45.91.116.11:4119 , piece_id=UOMKJYXLGJQR5IV23Z7XQ5BVVHU5IBODONKK4TE6UVSBR4MBN5LA, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.526620308 (195h45m21.54228991s) [noise=true , address=45.91.116.10:4081 , piece_id=QKUEKRSUKSRK5QUAKUIAYZKE3MVVTMJSOOI4UBRQ3OEVITPPUZ4A, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.567589144 (195h45m21.583258748s) [noise=true , address=79.127.219.45:4856 , piece_id=6YNPHXN42DTWSMURYVLWUHVNFYWLQWAWWKEEDTWQ43UNV7PXSBEQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.597278275 (195h45m21.612947879s) [noise=true , address=77.90.31.14:4663 , piece_id=ZHRSWG5OJZHAMHWIX5BJLHSFE5SDJEFRDWGFXFIQGPZWR3SLVJUA, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.598341351 (195h45m21.614010952s) [noise=true , address=77.90.31.16:4737 , piece_id=CHS4E5YFJ4E6773WCFLBRUFEISWY75EVQ46SUB7KIWZSDPJ3L4XA, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:18.606487466 (195h45m21.622157073s) [noise=true , address=77.90.31.18:4805 , piece_id=AYKQ2ZSEGER7UH5B7VNO34C5GQOTSJBO7SUTSV6PTGBQ5ZKTGWGQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: <nil> Mar 18 19:56:44.156578444 (195h45m47.172248053s) [noise=true , address=145.79.227.3:4155 , piece_id=4L42QGI32QXDUAY3RCE5TQUQZYYG4SSQB2GGSPJAJH6MXAMJVSAQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:145.79.227.3:4155): protocol: rpc: dial tcp 145.79.227.3:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:56:44.156649147 (195h45m47.172318758s) [noise=true , address=145.79.227.52:4448 , piece_id=6VQISZH4PYG7T67VBVQALI2K5FYHOBA63UVQWNZ752OHVS4V7KEQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: ecclient: upload failed (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i, address:145.79.227.52:4448): protocol: rpc: dial tcp 145.79.227.52:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:56:44.156851612 (195h45m47.172521213s) [noise=true , address=145.79.227.33:4874 , piece_id=TV6PQHDDRE4UNGWQDKTCAZERHO6LSN433LWIG4X244P3VKDEADYQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: ecclient: upload failed (node:12YApHMARaC7Wxeqopt72gpnyJS1xYBwhV7wbU9NJdR6W7hukyq, address:145.79.227.33:4874): protocol: rpc: dial tcp 145.79.227.33:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:56:44.157002699 (195h45m47.17267231s) [noise=true , address=145.79.227.44:4198 , piece_id=WX34JQXE2C2G3WCBO365G7NHNMA5AVXNL5LBL6LJSNNZRYLZPQSQ, seg_pos={0 0}, upload=188041]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:145.79.227.44:4198): protocol: rpc: dial tcp 145.79.227.44:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:56:44.157078200 (195h45m47.172747793s) [seg_pos={0 0}, upload=188041]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 19:56:44.259942651 (195h45m47.275612257s) [upload=188041]: Done uploading segments... Mar 18 19:56:44.260008753 (195h45m47.275678359s) [upload=188041]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.105940982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................X..X...X..X..X........X..X..] duration: 20.198402671s trace: https://jaeger.dev.storj.io/trace/79c41e16c37a125 FAIL (2026-03-18 19:56:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 38 (195h43m55.401637841s) [upload=188016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:54:52.385974208 (195h43m55.401643808s) [upload=188016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:54:56.485306613 (195h43m59.500976213s) [seg_pos={0 0}, upload=188016]: Begin upload segment... Mar 18 19:54:56.485373907 (195h43m59.501043508s) [seg_pos={0 0}, upload=188016]: Done begin upload segment. Mar 18 19:54:56.485413512 (195h43m59.501083122s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485484810 (195h43m59.501154413s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485835232 (195h43m59.501504841s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486319811 (195h43m59.501989413s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486707452 (195h43m59.502377054s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487087370 (195h43m59.502756983s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487167322 (195h43m59.502836933s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487527734 (195h43m59.503197345s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488222980 (195h43m59.503892583s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488692381 (195h43m59.504361987s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489068132 (195h43m59.504737735s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489428557 (195h43m59.505098166s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.490692212 (195h43m59.506361821s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.491637161 (195h43m59.507306765s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.492159147 (195h43m59.507828753s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.503552529 (195h43m59.519222129s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.506862549 (195h43m59.522532149s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.509361639 (195h43m59.525031239s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:57.052157823 (195h44m0.067827423s) [upload=188016]: Next returned nil segment Mar 18 19:54:57.052172349 (195h44m0.067841949s) [upload=188016]: Waiting for error group managing segments... Mar 18 19:54:57.192639690 (195h44m0.208309292s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.226146219 (195h44m0.241815822s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.282669419 (195h44m0.29833902s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.295703027 (195h44m0.311372629s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.335839459 (195h44m0.351509069s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.444257754 (195h44m0.459927354s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.578622592 (195h44m0.594292192s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.739657243 (195h44m0.755326843s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.748437301 (195h44m0.764106905s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.765988304 (195h44m0.781657917s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.900971239 (195h44m0.916640891s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:58.022122666 (195h44m1.037792267s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:55:52.098861860 (195h44m55.114531463s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:145.79.227.18:4032): protocol: rpc: dial tcp 145.79.227.18:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099047063 (195h44m55.114716663s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:145.79.227.49:4372): protocol: rpc: dial tcp 145.79.227.49:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099226684 (195h44m55.114896288s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:145.79.227.36:4523): protocol: rpc: dial tcp 145.79.227.36:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099406270 (195h44m55.11507587s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, address:145.79.227.40:4076): protocol: rpc: dial tcp 145.79.227.40:4076: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101466845 (195h44m55.117136454s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1mMFDDCJAQbesaWGwR5Z4cEvoN6eyNSZwCx4vc2HimniJzjwj7, address:145.79.227.38:4988): protocol: rpc: dial tcp 145.79.227.38:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101800994 (195h44m55.117470605s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ, address:145.79.227.22:4540): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:52.101848800 (195h44m55.117518391s) [seg_pos={0 0}, upload=188016]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 19:55:52.206680967 (195h44m55.222350568s) [upload=188016]: Done uploading segments... Mar 18 19:55:52.206716528 (195h44m55.222386128s) [upload=188016]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 20.198432651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..................................XXX.XXX...............X...] duration: 19.569745651s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:56:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.569769818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.............................XXXXXXXX....XX.XXXXXXXXXXXXXXXX] duration: 5.00097061s trace: https://jaeger.dev.storj.io/trace/777f69679e2728f7 FAIL (2026-03-18 19:56:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000992638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................X..X...X..X..X........X..X.] duration: 24.191507917s trace: https://jaeger.dev.storj.io/trace/79c41e16c37a125 FAIL (2026-03-18 19:56:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 38 (195h43m55.401637841s) [upload=188016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:54:52.385974208 (195h43m55.401643808s) [upload=188016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:54:56.485306613 (195h43m59.500976213s) [seg_pos={0 0}, upload=188016]: Begin upload segment... Mar 18 19:54:56.485373907 (195h43m59.501043508s) [seg_pos={0 0}, upload=188016]: Done begin upload segment. Mar 18 19:54:56.485413512 (195h43m59.501083122s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485484810 (195h43m59.501154413s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485835232 (195h43m59.501504841s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486319811 (195h43m59.501989413s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486707452 (195h43m59.502377054s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487087370 (195h43m59.502756983s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487167322 (195h43m59.502836933s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487527734 (195h43m59.503197345s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488222980 (195h43m59.503892583s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488692381 (195h43m59.504361987s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489068132 (195h43m59.504737735s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489428557 (195h43m59.505098166s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.490692212 (195h43m59.506361821s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.491637161 (195h43m59.507306765s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.492159147 (195h43m59.507828753s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.503552529 (195h43m59.519222129s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.506862549 (195h43m59.522532149s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.509361639 (195h43m59.525031239s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:57.052157823 (195h44m0.067827423s) [upload=188016]: Next returned nil segment Mar 18 19:54:57.052172349 (195h44m0.067841949s) [upload=188016]: Waiting for error group managing segments... Mar 18 19:54:57.192639690 (195h44m0.208309292s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.226146219 (195h44m0.241815822s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.282669419 (195h44m0.29833902s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.295703027 (195h44m0.311372629s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.335839459 (195h44m0.351509069s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.444257754 (195h44m0.459927354s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.578622592 (195h44m0.594292192s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.739657243 (195h44m0.755326843s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.748437301 (195h44m0.764106905s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.765988304 (195h44m0.781657917s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.900971239 (195h44m0.916640891s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:58.022122666 (195h44m1.037792267s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:55:52.098861860 (195h44m55.114531463s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:145.79.227.18:4032): protocol: rpc: dial tcp 145.79.227.18:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099047063 (195h44m55.114716663s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:145.79.227.49:4372): protocol: rpc: dial tcp 145.79.227.49:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099226684 (195h44m55.114896288s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:145.79.227.36:4523): protocol: rpc: dial tcp 145.79.227.36:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099406270 (195h44m55.11507587s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, address:145.79.227.40:4076): protocol: rpc: dial tcp 145.79.227.40:4076: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101466845 (195h44m55.117136454s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1mMFDDCJAQbesaWGwR5Z4cEvoN6eyNSZwCx4vc2HimniJzjwj7, address:145.79.227.38:4988): protocol: rpc: dial tcp 145.79.227.38:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101800994 (195h44m55.117470605s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ, address:145.79.227.22:4540): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:52.101848800 (195h44m55.117518391s) [seg_pos={0 0}, upload=188016]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 19:55:52.206680967 (195h44m55.222350568s) [upload=188016]: Done uploading segments... Mar 18 19:55:52.206716528 (195h44m55.222386128s) [upload=188016]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 24.191526732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...................................XXX.XXX...............X..] duration: 18.948313291s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:56:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.948338592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................X..X...X..X..X........X..X] duration: 1m0.110564583s trace: https://jaeger.dev.storj.io/trace/79c41e16c37a125 FAIL (2026-03-18 19:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 38 (195h43m55.401637841s) [upload=188016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:54:52.385974208 (195h43m55.401643808s) [upload=188016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:54:56.485306613 (195h43m59.500976213s) [seg_pos={0 0}, upload=188016]: Begin upload segment... Mar 18 19:54:56.485373907 (195h43m59.501043508s) [seg_pos={0 0}, upload=188016]: Done begin upload segment. Mar 18 19:54:56.485413512 (195h43m59.501083122s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485484810 (195h43m59.501154413s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.485835232 (195h43m59.501504841s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486319811 (195h43m59.501989413s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.486707452 (195h43m59.502377054s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487087370 (195h43m59.502756983s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487167322 (195h43m59.502836933s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.487527734 (195h43m59.503197345s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488222980 (195h43m59.503892583s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.488692381 (195h43m59.504361987s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489068132 (195h43m59.504737735s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.489428557 (195h43m59.505098166s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.490692212 (195h43m59.506361821s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.491637161 (195h43m59.507306765s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.492159147 (195h43m59.507828753s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.503552529 (195h43m59.519222129s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.506862549 (195h43m59.522532149s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:56.509361639 (195h43m59.525031239s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Uploading piece... Mar 18 19:54:57.052157823 (195h44m0.067827423s) [upload=188016]: Next returned nil segment Mar 18 19:54:57.052172349 (195h44m0.067841949s) [upload=188016]: Waiting for error group managing segments... Mar 18 19:54:57.192639690 (195h44m0.208309292s) [noise=true , address=79.127.219.46:4895 , piece_id=OWW5DXTINCK6A4ANI4HIQTK4HKXWP7UKUBFRHF7KAKOLSX5URVDQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.226146219 (195h44m0.241815822s) [noise=true , address=79.127.219.38:4834 , piece_id=S2NUIFY3W7SVMNEHFBGNZDW3XNIBS4OXNK2T7V5AFQZANUTCXNFA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.282669419 (195h44m0.29833902s) [noise=true , address=79.127.219.45:4866 , piece_id=3U7TSD5QDQYKFQXPZFX2IOZKKCACGVN5MQ22HKV3F3RDD2CHGREA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.295703027 (195h44m0.311372629s) [noise=true , address=45.91.116.5:4303 , piece_id=OV2E2T2LI4V3HXXMQZ6F2CG3DAK6QQFUT2HAGDKMZQWLFSPTJRJQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.335839459 (195h44m0.351509069s) [noise=true , address=77.90.31.21:4905 , piece_id=C3Y2DUMWWII3HW2UTN7N45E6BQOGNEWNELWNK3XWO3FNYCLVPWQQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.444257754 (195h44m0.459927354s) [noise=true , address=77.90.31.16:4760 , piece_id=CCAM2QFGWKREOZDAMBE5ZGCY2RT4HATNUKIZOPTEO4UHLUF4RODA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.578622592 (195h44m0.594292192s) [noise=true , address=167.17.72.70:4780 , piece_id=H5UXJTZEXEGXPT6EGQRCM5DYAAF5MG6A7L5TFG334ZMSNR2VP3KQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.739657243 (195h44m0.755326843s) [noise=true , address=193.149.164.71:4305 , piece_id=4UMOBLMJRU2M4T52QQNYE3AKM7V6Y2MDQ526HWODJADQUJWYUVMA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.748437301 (195h44m0.764106905s) [noise=true , address=193.149.164.70:4135 , piece_id=CM7CKQHT5NPIRTK6SFIZYTCGVMWFV6GFU6K4OYYGWYB4ISQZM24A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.765988304 (195h44m0.781657917s) [noise=true , address=193.149.164.67:4029 , piece_id=W432QK7EW26SVERGNS3SG3FVQL3YRQYUQ7BZIKBM332QC7R4GLHA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:57.900971239 (195h44m0.916640891s) [noise=true , address=79.127.205.233:4146 , piece_id=UC6JS2LYABONFBT54WLKZTOMOGVN4LWN5H4YKKKHAJKXPQ4NACVA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:54:58.022122666 (195h44m1.037792267s) [noise=true , address=79.127.205.237:4232 , piece_id=BGOJEBDKE6QIIFAMIDWPENIRDYMJLAGYTTVHH7RLPJ36J5QCFBHQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: <nil> Mar 18 19:55:52.098861860 (195h44m55.114531463s) [noise=true , address=145.79.227.18:4032 , piece_id=4G5D2EJGQLPTSWDOLREIRDVAW23NHSTGRBFSQH632DVPLQCRJN3A, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:145.79.227.18:4032): protocol: rpc: dial tcp 145.79.227.18:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099047063 (195h44m55.114716663s) [noise=true , address=145.79.227.49:4372 , piece_id=66MHGLXJWXPONP32WBTLAUW5GXPAV4FLCTOCA32HES2U5WRO6ERQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:145.79.227.49:4372): protocol: rpc: dial tcp 145.79.227.49:4372: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099226684 (195h44m55.114896288s) [noise=true , address=145.79.227.36:4523 , piece_id=2KWHTQHRMAOYFAEVOEOC6V7CC7GUHRNJY2PDXTQ3MK7AC5NVJZYQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:145.79.227.36:4523): protocol: rpc: dial tcp 145.79.227.36:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.099406270 (195h44m55.11507587s) [noise=true , address=145.79.227.40:4076 , piece_id=3LIR3STQKO4ZAVUTVQ4QQV572LUNV2E2V6CSPRNGDKWK663DI7GA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2, address:145.79.227.40:4076): protocol: rpc: dial tcp 145.79.227.40:4076: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101466845 (195h44m55.117136454s) [noise=true , address=145.79.227.38:4988 , piece_id=F67CSA67BC73EMIIF67DBFRPNHKOX3BPL7WBBY7VRUVOJP6JYZZQ, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1mMFDDCJAQbesaWGwR5Z4cEvoN6eyNSZwCx4vc2HimniJzjwj7, address:145.79.227.38:4988): protocol: rpc: dial tcp 145.79.227.38:4988: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:52.101800994 (195h44m55.117470605s) [noise=true , address=145.79.227.22:4540 , piece_id=BXETISY3ROPOMCOZY5CW3WEZWZPKB2OGLKSFMSTN2CXIFKOHFBKA, seg_pos={0 0}, upload=188016]: Done uploading piece. err: ecclient: upload failed (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ, address:145.79.227.22:4540): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:52.101848800 (195h44m55.117518391s) [seg_pos={0 0}, upload=188016]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 18 19:55:52.206680967 (195h44m55.222350568s) [upload=188016]: Done uploading segments... Mar 18 19:55:52.206716528 (195h44m55.222386128s) [upload=188016]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.110589453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.127864866s trace: https://jaeger.dev.storj.io/trace/58b836bc7d252230 FAIL (2026-03-18 19:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 4) ======LOGS====== 4882 , piece_id=Z4UAME3W2GA4HCA5AYZSKR46WSPOTDSONNGWCPJIWRWAAKFAPJ6Q, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.939934215 (195h44m12.955603822s) [noise=true , address=77.90.31.18:4809 , piece_id=RDNSJSQWTPA3BBZHQFJGCBK25PAIFWNJJRTTUZFEBSYOK36LVSDA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.940516519 (195h44m12.956186123s) [noise=true , address=145.79.227.19:4081 , piece_id=H7ROV2QQTU5QSL6NLNYURQAFYXU7TOEMZMHFRXBTK4W5DX2N7VFQ, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.940584131 (195h44m12.956253741s) [noise=true , address=145.79.227.50:4374 , piece_id=B33J45WT6JMRKFQAFUHS6M2AA2RVWDBLVEII3UNQOGLZ2ZIYFC3Q, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.940976837 (195h44m12.956646437s) [noise=true , address=45.91.116.5:4301 , piece_id=GKRIKPV3N42YPG5HOTL44B5RD7GUJLLETYQOM67EYWMEHLI52HAA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.941122448 (195h44m12.956792066s) [noise=true , address=145.79.227.13:4807 , piece_id=RSV4ZENDBMT3HFTFCHZSZTD6W3W37FD65MSTWGGP77PGSH7KMZLA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.941314729 (195h44m12.956984332s) [noise=true , address=145.79.227.29:4946 , piece_id=MBBF3H2OKVG4CRNORS4HMWB3ILRSVDQQZBPKJJMRSFGUALK75FKQ, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.941851287 (195h44m12.95752089s) [noise=true , address=45.91.116.6:4976 , piece_id=A3KFMHY3BMG7MDNCGNGUQG4JF4ES6AW6FPTO7PAMDMG6UKQJQ64A, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.941906467 (195h44m12.957576089s) [noise=true , address=145.79.227.42:4123 , piece_id=USL2DCAKQFBYUJ55AXLL6FT7LEBLM6RHLVNLLD4Q52HPZ3WLZL6A, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.942271995 (195h44m12.957941598s) [noise=true , address=145.79.227.3:4149 , piece_id=TNTHJSQAQ7WGE3IZR3GB447WIHZNGWOUXA5WZDS3S2ECYLUXPOUA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.942566404 (195h44m12.958236015s) [noise=true , address=145.79.227.52:4447 , piece_id=VXZFLQ446YZMNKFV3DW55DEDETZFBHTMM4YOX7MHCOPM4B5CFJMA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.942608680 (195h44m12.958278281s) [noise=true , address=145.79.227.36:4517 , piece_id=QDQINESEHUCI2IETUSYFQN2QEPKEWBINHSZQZZTXLAC5DTRW2ZLA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.942967761 (195h44m12.958637369s) [noise=true , address=145.79.227.49:4342 , piece_id=OGK3OA3U5LWUFLCT3FGYK24LU43QY6BOQU5XI7PZJ74BLKVSS2UQ, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.943173562 (195h44m12.958843177s) [noise=true , address=145.79.227.10:4406 , piece_id=H4AVE7YD6VWO55TY6ATCML7MGJAXWEQ2HESB5ROAJDCMIMGHI2UA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.943832395 (195h44m12.959502006s) [noise=true , address=145.79.227.41:4085 , piece_id=HJQKMFJC6PI2JW7GUQ4W5N4I7G6GF2DJ2U5ABQX7SAEAVZ2IVHQA, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:09.944472103 (195h44m12.960141712s) [noise=true , address=145.79.227.5:4221 , piece_id=T53Y6P6B4OLXRAW6LYUEIA7E7FTITFJJ4EJO7TMIHQTBERZGZVTQ, seg_pos={0 0}, upload=188024]: Uploading piece... Mar 18 19:55:10.582210435 (195h44m13.597880025s) [upload=188024]: Next returned nil segment Mar 18 19:55:10.582224542 (195h44m13.597894142s) [upload=188024]: Waiting for error group managing segments... Mar 18 19:55:10.688281075 (195h44m13.70395069s) [noise=true , address=45.91.116.5:4301 , piece_id=GKRIKPV3N42YPG5HOTL44B5RD7GUJLLETYQOM67EYWMEHLI52HAA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: <nil> Mar 18 19:55:10.713253337 (195h44m13.728922941s) [noise=true , address=45.91.116.6:4976 , piece_id=A3KFMHY3BMG7MDNCGNGUQG4JF4ES6AW6FPTO7PAMDMG6UKQJQ64A, seg_pos={0 0}, upload=188024]: Done uploading piece. err: <nil> Mar 18 19:55:10.755628122 (195h44m13.771297729s) [noise=true , address=77.90.31.21:4882 , piece_id=Z4UAME3W2GA4HCA5AYZSKR46WSPOTDSONNGWCPJIWRWAAKFAPJ6Q, seg_pos={0 0}, upload=188024]: Done uploading piece. err: <nil> Mar 18 19:55:10.767177556 (195h44m13.782847155s) [noise=true , address=77.90.31.18:4809 , piece_id=RDNSJSQWTPA3BBZHQFJGCBK25PAIFWNJJRTTUZFEBSYOK36LVSDA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: <nil> Mar 18 19:55:36.930622688 (195h44m39.946292325s) [noise=true , address=145.79.227.41:4085 , piece_id=HJQKMFJC6PI2JW7GUQ4W5N4I7G6GF2DJ2U5ABQX7SAEAVZ2IVHQA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:145.79.227.41:4085): protocol: rpc: dial tcp 145.79.227.41:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.930677508 (195h44m39.946347114s) [noise=true , address=145.79.227.3:4149 , piece_id=TNTHJSQAQ7WGE3IZR3GB447WIHZNGWOUXA5WZDS3S2ECYLUXPOUA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.930939540 (195h44m39.946609152s) [noise=true , address=145.79.227.5:4221 , piece_id=T53Y6P6B4OLXRAW6LYUEIA7E7FTITFJJ4EJO7TMIHQTBERZGZVTQ, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931015401 (195h44m39.946685013s) [noise=true , address=145.79.227.19:4081 , piece_id=H7ROV2QQTU5QSL6NLNYURQAFYXU7TOEMZMHFRXBTK4W5DX2N7VFQ, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:145.79.227.19:4081): protocol: rpc: dial tcp 145.79.227.19:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931073324 (195h44m39.946742938s) [noise=true , address=145.79.227.13:4807 , piece_id=RSV4ZENDBMT3HFTFCHZSZTD6W3W37FD65MSTWGGP77PGSH7KMZLA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:145.79.227.13:4807): protocol: rpc: dial tcp 145.79.227.13:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931249104 (195h44m39.946918708s) [noise=true , address=145.79.227.36:4517 , piece_id=QDQINESEHUCI2IETUSYFQN2QEPKEWBINHSZQZZTXLAC5DTRW2ZLA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:145.79.227.36:4517): protocol: rpc: dial tcp 145.79.227.36:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931312726 (195h44m39.946982336s) [noise=true , address=145.79.227.10:4406 , piece_id=H4AVE7YD6VWO55TY6ATCML7MGJAXWEQ2HESB5ROAJDCMIMGHI2UA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:1KrucHjbKjhBZEUGd2p8cmfyCwm6rx4BVwwVhkpgnEv9EQiJRB, address:145.79.227.10:4406): protocol: rpc: dial tcp 145.79.227.10:4406: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931413873 (195h44m39.947083478s) [noise=true , address=145.79.227.29:4946 , piece_id=MBBF3H2OKVG4CRNORS4HMWB3ILRSVDQQZBPKJJMRSFGUALK75FKQ, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX, address:145.79.227.29:4946): protocol: rpc: dial tcp 145.79.227.29:4946: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.931695015 (195h44m39.947364623s) [noise=true , address=145.79.227.52:4447 , piece_id=VXZFLQ446YZMNKFV3DW55DEDETZFBHTMM4YOX7MHCOPM4B5CFJMA, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: rpc: dial tcp 145.79.227.52:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.932205127 (195h44m39.94787474s) [noise=true , address=145.79.227.50:4374 , piece_id=B33J45WT6JMRKFQAFUHS6M2AA2RVWDBLVEII3UNQOGLZ2ZIYFC3Q, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.932456145 (195h44m39.948125755s) [noise=true , address=145.79.227.49:4342 , piece_id=OGK3OA3U5LWUFLCT3FGYK24LU43QY6BOQU5XI7PZJ74BLKVSS2UQ, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:145.79.227.49:4342): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:36.932777794 (195h44m39.948447404s) [noise=true , address=145.79.227.42:4123 , piece_id=USL2DCAKQFBYUJ55AXLL6FT7LEBLM6RHLVNLLD4Q52HPZ3WLZL6A, seg_pos={0 0}, upload=188024]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:36.932816300 (195h44m39.948485887s) [seg_pos={0 0}, upload=188024]: Done waiting for segment. successful: 4 optimal: 13 errs: <nil> Mar 18 19:55:37.056735232 (195h44m40.072404824s) [upload=188024]: Done uploading segments... Mar 18 19:55:37.056771923 (195h44m40.072441529s) [upload=188024]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.127886672s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....................................XXX.XXX...............X.] duration: 19.240884396s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:55:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.241048547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..............................XXXXXXXX....XX.XXXXXXXXXXXXXXX] duration: 5.000726509s trace: https://jaeger.dev.storj.io/trace/6636a4c71d2f88e5 FAIL (2026-03-18 19:55:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000759579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.112650232s trace: https://jaeger.dev.storj.io/trace/46fa623bbf285ca7 FAIL (2026-03-18 19:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== Mar 18 19:54:34.882474739 (195h43m37.898144339s) [upload=188012]: Starting upload Mar 18 19:54:34.882609951 (195h43m37.898279538s) [upload=188012]: Done starting upload Mar 18 19:54:34.882653908 (195h43m37.898323494s) [upload=188012]: Uploading segments... Mar 18 19:54:34.883029255 (195h43m37.898698841s) [upload=188012]: Got next segment. Inline: false Mar 18 19:54:34.883081060 (195h43m37.89875066s) [upload=188012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:54:34.883085451 (195h43m37.898755051s) [upload=188012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:54:37.644659637 (195h43m40.660329225s) [seg_pos={0 0}, upload=188012]: Begin upload segment... Mar 18 19:54:37.644723638 (195h43m40.660393228s) [seg_pos={0 0}, upload=188012]: Done begin upload segment. Mar 18 19:54:37.644756069 (195h43m40.660425673s) [noise=true , address=45.91.116.6:4973 , piece_id=JLSRAA4NRPRNALLL22X7XN4GTPEM7IG6GDSTJZ4SALXQFLLBBMBQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.644819627 (195h43m40.660489233s) [noise=true , address=45.91.116.12:4225 , piece_id=WV46DOKTOLUL346ODMCQJQYSHW5KLZHAGZDYLEJXYQUDJSOVTYUQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.645417197 (195h43m40.661086804s) [noise=true , address=145.79.227.7:4299 , piece_id=3CG5L2P6A45TU4VG6JUXT2ONE2TCZNBSFRUB7G3WGGWPWXAFFLHA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.646119386 (195h43m40.661788996s) [noise=true , address=145.79.227.35:4484 , piece_id=O6EF2NNUEI2RQ52J4AM3H3KFRFLDPYGIALGLL5XIKRAGQIMJNOYQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.646678417 (195h43m40.66234802s) [noise=true , address=145.79.227.50:4377 , piece_id=K4ENFDN6EG2ECTXDLO5TOXLJN2F4YH3QRZRJEQPJ2V2ZRZR652QQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.647124593 (195h43m40.662794197s) [noise=true , address=77.90.31.13:4625 , piece_id=PJHFTPXBLTZATUJENZVCBFAJ6OMUWFUWPKHSM37O4OJMGKGONAPQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.647568608 (195h43m40.663238208s) [noise=true , address=145.79.227.40:4049 , piece_id=GMO2FMQ5W45FAMRPUHX4FOJ4UQ3WD6CIVNOCNS3J7ASNQR5MIWIA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.647885909 (195h43m40.663555513s) [noise=true , address=145.79.227.20:4441 , piece_id=6FQQNPZSA5MJAGWENKNUPGDCNVQ7PWKEFG2ADDWYSRF2MQAT7MGQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.648382335 (195h43m40.664051937s) [noise=true , address=145.79.227.2:4113 , piece_id=B4PIELZVT4TD2R7H3LYHXJ2AJU3QUM3GCR66BYWQUYP6AA4NPTJA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.648960871 (195h43m40.664630474s) [noise=true , address=145.79.227.31:4801 , piece_id=SVWRMSAZHI72LNBOMBM4IHAKGNMDFRPWPEROHG3YTSA4YDFBSH4Q, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.649510760 (195h43m40.665180362s) [noise=true , address=45.91.116.11:4118 , piece_id=MGAHEVQWTRCHUC6JXGYAYBLQCFL2AB3JWFJTQ4BCG4DWOHNIG5BA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.650186482 (195h43m40.665856082s) [noise=true , address=145.79.227.3:4149 , piece_id=4ESUNPTWSFACCOBLE5SVDEXLDIQTTXWCGKU5WKXB4NKB4WOPMDEQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.650209701 (195h43m40.665879306s) [noise=true , address=77.90.31.11:4558 , piece_id=MGGGYQLI7KQFV2NVGTCXZTM5UL5ZFWKGJSH7SL4QJDSO6XCHUKBA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.650711514 (195h43m40.666381117s) [noise=true , address=45.91.116.5:4297 , piece_id=JZ27F7A45SFN3OBPYJLLJPZVC2336ITNPINWSEJ5WVEGMR7IT7NQ, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.650743072 (195h43m40.666412698s) [noise=true , address=45.91.116.10:4083 , piece_id=KQNQWR3ETI27F4HBOWVTJU72ACR565XZU3TIOFVLPL3OK4VNRNNA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:37.651259872 (195h43m40.666929478s) [noise=true , address=145.79.227.52:4447 , piece_id=QVLI44XKQTE6A7QMLFD6WOH3DGWF7HAM3FARHD37JGDYVDPAOVGA, seg_pos={0 0}, upload=188012]: Uploading piece... Mar 18 19:54:38.244728268 (195h43m41.260397856s) [upload=188012]: Next returned nil segment Mar 18 19:54:38.244746208 (195h43m41.260415826s) [upload=188012]: Waiting for error group managing segments... Mar 18 19:54:38.382932254 (195h43m41.398601856s) [noise=true , address=77.90.31.11:4558 , piece_id=MGGGYQLI7KQFV2NVGTCXZTM5UL5ZFWKGJSH7SL4QJDSO6XCHUKBA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.386798585 (195h43m41.402468186s) [noise=true , address=45.91.116.11:4118 , piece_id=MGAHEVQWTRCHUC6JXGYAYBLQCFL2AB3JWFJTQ4BCG4DWOHNIG5BA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.391858137 (195h43m41.40752774s) [noise=true , address=45.91.116.5:4297 , piece_id=JZ27F7A45SFN3OBPYJLLJPZVC2336ITNPINWSEJ5WVEGMR7IT7NQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.405134317 (195h43m41.420803925s) [noise=true , address=45.91.116.6:4973 , piece_id=JLSRAA4NRPRNALLL22X7XN4GTPEM7IG6GDSTJZ4SALXQFLLBBMBQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.406319619 (195h43m41.421989228s) [noise=true , address=45.91.116.10:4083 , piece_id=KQNQWR3ETI27F4HBOWVTJU72ACR565XZU3TIOFVLPL3OK4VNRNNA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.423245270 (195h43m41.43891487s) [noise=true , address=45.91.116.12:4225 , piece_id=WV46DOKTOLUL346ODMCQJQYSHW5KLZHAGZDYLEJXYQUDJSOVTYUQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:54:38.458449626 (195h43m41.474119226s) [noise=true , address=77.90.31.13:4625 , piece_id=PJHFTPXBLTZATUJENZVCBFAJ6OMUWFUWPKHSM37O4OJMGKGONAPQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: <nil> Mar 18 19:55:04.588577114 (195h44m7.604246724s) [noise=true , address=145.79.227.50:4377 , piece_id=K4ENFDN6EG2ECTXDLO5TOXLJN2F4YH3QRZRJEQPJ2V2ZRZR652QQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:126HKpC7sciWFTxNaH5Hjs6BBuh8NRgu8fA4cu4qcK29zfJWg7R, address:145.79.227.50:4377): protocol: rpc: dial tcp 145.79.227.50:4377: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:04.588940674 (195h44m7.604610285s) [noise=true , address=145.79.227.35:4484 , piece_id=O6EF2NNUEI2RQ52J4AM3H3KFRFLDPYGIALGLL5XIKRAGQIMJNOYQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:145.79.227.35:4484): protocol: rpc: dial tcp 145.79.227.35:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:04.589162798 (195h44m7.604832407s) [noise=true , address=145.79.227.7:4299 , piece_id=3CG5L2P6A45TU4VG6JUXT2ONE2TCZNBSFRUB7G3WGGWPWXAFFLHA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:12sX7515Lspa6uoZAfWzqPhmnMPcvrVaASSPungFuZEBhmApN9n, address:145.79.227.7:4299): protocol: rpc: dial tcp 145.79.227.7:4299: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:55:04.590315456 (195h44m7.605985284s) [noise=true , address=145.79.227.3:4149 , piece_id=4ESUNPTWSFACCOBLE5SVDEXLDIQTTXWCGKU5WKXB4NKB4WOPMDEQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590487478 (195h44m7.606157091s) [noise=true , address=145.79.227.52:4447 , piece_id=QVLI44XKQTE6A7QMLFD6WOH3DGWF7HAM3FARHD37JGDYVDPAOVGA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590519164 (195h44m7.60618882s) [noise=true , address=145.79.227.40:4049 , piece_id=GMO2FMQ5W45FAMRPUHX4FOJ4UQ3WD6CIVNOCNS3J7ASNQR5MIWIA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590682259 (195h44m7.606351861s) [noise=true , address=145.79.227.2:4113 , piece_id=B4PIELZVT4TD2R7H3LYHXJ2AJU3QUM3GCR66BYWQUYP6AA4NPTJA, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:145.79.227.2:4113): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590696826 (195h44m7.606366436s) [noise=true , address=145.79.227.20:4441 , piece_id=6FQQNPZSA5MJAGWENKNUPGDCNVQ7PWKEFG2ADDWYSRF2MQAT7MGQ, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590881781 (195h44m7.606551392s) [noise=true , address=145.79.227.31:4801 , piece_id=SVWRMSAZHI72LNBOMBM4IHAKGNMDFRPWPEROHG3YTSA4YDFBSH4Q, seg_pos={0 0}, upload=188012]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:55:04.590926558 (195h44m7.60659616s) [seg_pos={0 0}, upload=188012]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 19:55:04.698740582 (195h44m7.714410193s) [upload=188012]: Done uploading segments... Mar 18 19:55:04.698786551 (195h44m7.714456159s) [upload=188012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.11266888s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....................................XXX.XXX...............X] duration: 1m0.000751316s trace: https://jaeger.dev.storj.io/trace/15e13da99079fd26 FAIL (2026-03-18 19:55:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000848965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...............................XXXXXXXX....XX.XXXXXXXXXXXXXX] duration: 5.000730181s trace: https://jaeger.dev.storj.io/trace/18deb36caf2a9627 FAIL (2026-03-18 19:54:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000753283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.109888012s trace: https://jaeger.dev.storj.io/trace/267dfd95bcbc2dc0 FAIL (2026-03-18 19:54:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== .612518790 (195h43m7.628188402s) [upload=188004]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:54:08.814946237 (195h43m11.830615829s) [seg_pos={0 0}, upload=188004]: Begin upload segment... Mar 18 19:54:08.815167481 (195h43m11.830837069s) [seg_pos={0 0}, upload=188004]: Done begin upload segment. Mar 18 19:54:08.815246532 (195h43m11.830916136s) [noise=true , address=45.91.116.8:4046 , piece_id=YPYNO6B56XURLPS2OPODWJ7EIQVQXOJRFZ7EAWODN6FABNXPYYCA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.816249095 (195h43m11.831918695s) [noise=true , address=145.79.227.27:4692 , piece_id=MNAS7ZEIQWVFJ2LK4VPNRMVKXU4APDQRHVYWMHCGTO6C2KRNVK7Q, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.816349365 (195h43m11.832018975s) [noise=true , address=145.79.227.26:4659 , piece_id=FZ4NQD3JEGRQXQW4K3M37QTIG6EYUW7MFT3IK5X7XQLYANJIJVSQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.816798277 (195h43m11.832467884s) [noise=true , address=145.79.227.11:4735 , piece_id=5BWPTTAGB5V3RRLC43LJRJYFM4FZ4NMFDKNGKL4W67P4CVRMO7BQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.816899445 (195h43m11.832569055s) [noise=true , address=77.90.31.21:4878 , piece_id=TI5Q5VAN3TV4YTQQIAYWIMZYEEBJBOH6LFJQV4WTCPE77DLWUMDA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.817307485 (195h43m11.832977096s) [noise=true , address=145.79.227.20:4441 , piece_id=K2JHNMRQWO4L3O7P2PUCEUEA3WF4EOLFAZFZJILCIIXZX5N2IHYQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.817389018 (195h43m11.833058625s) [noise=true , address=145.79.227.9:4369 , piece_id=P7W7OAS7VNJMEOCKBNW5CUONTUPEHWK3IEXFZJACFP3TG3N5QAWA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.818036691 (195h43m11.833706302s) [noise=true , address=145.79.227.19:4078 , piece_id=KTYRGUYEJJBSYRWAEZ6YZWG3354LVXSOY6HQPU5QKOU7OON7AIDA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.818056276 (195h43m11.833725878s) [noise=true , address=45.91.116.14:4224 , piece_id=VHPFGNJZCMOBXWL3M6CMU2IQSTQNC2O4X4XXBGXOKX3AAVQ2FMLQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.818492399 (195h43m11.834162001s) [noise=true , address=145.79.227.3:4149 , piece_id=HEL7SPQYO7Q43TZWJIIY4LHYQ4ZI7AVCO7PO2H3GLE2JAXB5BBFQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.818529674 (195h43m11.834199284s) [noise=true , address=45.91.116.9:4261 , piece_id=XN3IGAAHHBJS7SHQ4F3RX4OZR46YPDH3HR7BAMQ4LSM5FORWFVQQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.819016402 (195h43m11.834685994s) [noise=true , address=145.79.227.47:4266 , piece_id=YFR2L6VNTNWJGWAZ2CZ7HM33W6UJWLLF3QMX4M53IOKELU7C6CCA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.819026277 (195h43m11.834695886s) [noise=true , address=45.91.116.11:4118 , piece_id=MHRURCUY5CPKF7ZDGSO2CM4NNDUQFYGVHKUQPSSBKL3WZCKCOOHQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.819529107 (195h43m11.835198712s) [noise=true , address=145.79.227.34:4909 , piece_id=RSRP24HZDPREXIYSFKDMNLMFYUIQGPWAVK4O36IKYDUJ4YRSQY7Q, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.820909305 (195h43m11.836578894s) [noise=true , address=145.79.227.25:4622 , piece_id=5X6C7K7RY6QIGXLTGQLUPCMQUOASSRA2FVHL2GOQOFTINKZNYMIQ, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:08.821258342 (195h43m11.836927945s) [noise=true , address=79.127.219.38:4819 , piece_id=YN6JDBGJF4JMPO45JX4GVEJKRCI5LMDHZYTSHYRDNZB26OJLBFUA, seg_pos={0 0}, upload=188004]: Uploading piece... Mar 18 19:54:09.423506577 (195h43m12.439176163s) [upload=188004]: Next returned nil segment Mar 18 19:54:09.423526998 (195h43m12.439196608s) [upload=188004]: Waiting for error group managing segments... Mar 18 19:54:09.554843530 (195h43m12.570513134s) [noise=true , address=77.90.31.21:4878 , piece_id=TI5Q5VAN3TV4YTQQIAYWIMZYEEBJBOH6LFJQV4WTCPE77DLWUMDA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:09.564877430 (195h43m12.58054703s) [noise=true , address=45.91.116.11:4118 , piece_id=MHRURCUY5CPKF7ZDGSO2CM4NNDUQFYGVHKUQPSSBKL3WZCKCOOHQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:09.566041188 (195h43m12.581710781s) [noise=true , address=45.91.116.9:4261 , piece_id=XN3IGAAHHBJS7SHQ4F3RX4OZR46YPDH3HR7BAMQ4LSM5FORWFVQQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:09.566622349 (195h43m12.582291954s) [noise=true , address=45.91.116.8:4046 , piece_id=YPYNO6B56XURLPS2OPODWJ7EIQVQXOJRFZ7EAWODN6FABNXPYYCA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:09.567440780 (195h43m12.583110373s) [noise=true , address=45.91.116.14:4224 , piece_id=VHPFGNJZCMOBXWL3M6CMU2IQSTQNC2O4X4XXBGXOKX3AAVQ2FMLQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:09.619134276 (195h43m12.634803863s) [noise=true , address=79.127.219.38:4819 , piece_id=YN6JDBGJF4JMPO45JX4GVEJKRCI5LMDHZYTSHYRDNZB26OJLBFUA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: <nil> Mar 18 19:54:34.330586577 (195h43m37.346256204s) [noise=true , address=145.79.227.34:4909 , piece_id=RSRP24HZDPREXIYSFKDMNLMFYUIQGPWAVK4O36IKYDUJ4YRSQY7Q, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh, address:145.79.227.34:4909): protocol: rpc: dial tcp 145.79.227.34:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.330888137 (195h43m37.346557769s) [noise=true , address=145.79.227.3:4149 , piece_id=HEL7SPQYO7Q43TZWJIIY4LHYQ4ZI7AVCO7PO2H3GLE2JAXB5BBFQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.331242881 (195h43m37.34691249s) [noise=true , address=145.79.227.20:4441 , piece_id=K2JHNMRQWO4L3O7P2PUCEUEA3WF4EOLFAZFZJILCIIXZX5N2IHYQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.332231351 (195h43m37.347900971s) [noise=true , address=145.79.227.47:4266 , piece_id=YFR2L6VNTNWJGWAZ2CZ7HM33W6UJWLLF3QMX4M53IOKELU7C6CCA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:145.79.227.47:4266): protocol: rpc: dial tcp 145.79.227.47:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.332516837 (195h43m37.348186448s) [noise=true , address=145.79.227.26:4659 , piece_id=FZ4NQD3JEGRQXQW4K3M37QTIG6EYUW7MFT3IK5X7XQLYANJIJVSQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:12UBuWLogtN8mv4nwXDpzGdQtbKDz1EfVENSjXC54QdFp2fJg99, address:145.79.227.26:4659): protocol: rpc: dial tcp 145.79.227.26:4659: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.332749307 (195h43m37.348418916s) [noise=true , address=145.79.227.9:4369 , piece_id=P7W7OAS7VNJMEOCKBNW5CUONTUPEHWK3IEXFZJACFP3TG3N5QAWA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: rpc: dial tcp 145.79.227.9:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.333218543 (195h43m37.348888149s) [noise=true , address=145.79.227.27:4692 , piece_id=MNAS7ZEIQWVFJ2LK4VPNRMVKXU4APDQRHVYWMHCGTO6C2KRNVK7Q, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.333553357 (195h43m37.349222966s) [noise=true , address=145.79.227.11:4735 , piece_id=5BWPTTAGB5V3RRLC43LJRJYFM4FZ4NMFDKNGKL4W67P4CVRMO7BQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:125ShGXckiJ3N5UWVmsMdhj98oFaRN7sg1KRucyUzqENW4Jp4kS, address:145.79.227.11:4735): protocol: rpc: dial tcp 145.79.227.11:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.333788191 (195h43m37.3494578s) [noise=true , address=145.79.227.19:4078 , piece_id=KTYRGUYEJJBSYRWAEZ6YZWG3354LVXSOY6HQPU5QKOU7OON7AIDA, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.334302808 (195h43m37.349972449s) [noise=true , address=145.79.227.25:4622 , piece_id=5X6C7K7RY6QIGXLTGQLUPCMQUOASSRA2FVHL2GOQOFTINKZNYMIQ, seg_pos={0 0}, upload=188004]: Done uploading piece. err: ecclient: upload failed (node:1xEK5XL3sYcip7PsQcZBQYihgnEZykdMmsKkCr2TUmbTFYncfs, address:145.79.227.25:4622): protocol: rpc: dial tcp 145.79.227.25:4622: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:54:34.334351162 (195h43m37.350020752s) [seg_pos={0 0}, upload=188004]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Mar 18 19:54:34.438168741 (195h43m37.453838332s) [upload=188004]: Done uploading segments... Mar 18 19:54:34.438226831 (195h43m37.453896441s) [upload=188004]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.109913143s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [................................XXXXXXXX....XX.XXXXXXXXXXXXX] duration: 5.001468829s trace: https://jaeger.dev.storj.io/trace/6655b24d2fd9f6a0 FAIL (2026-03-18 19:54:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001499748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................X..X...X..X..X........X..] duration: 19.353438529s trace: https://jaeger.dev.storj.io/trace/20c3e3002cfafbf1 FAIL (2026-03-18 19:53:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== tarting upload Mar 18 19:51:51.422439741 (195h40m54.43810934s) [upload=187967]: Uploading segments... Mar 18 19:51:51.422756046 (195h40m54.438425655s) [upload=187967]: Got next segment. Inline: false Mar 18 19:51:51.422805418 (195h40m54.438475023s) [upload=187967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:51:51.422814967 (195h40m54.438484569s) [upload=187967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:51:55.535636400 (195h40m58.551306s) [seg_pos={0 0}, upload=187967]: Begin upload segment... Mar 18 19:51:55.535747846 (195h40m58.551417436s) [seg_pos={0 0}, upload=187967]: Done begin upload segment. Mar 18 19:51:55.535781095 (195h40m58.551450701s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.535809704 (195h40m58.551479313s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536014863 (195h40m58.551684463s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536516173 (195h40m58.552185775s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536952649 (195h40m58.55262225s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537353909 (195h40m58.553023508s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537775660 (195h40m58.55344528s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538164137 (195h40m58.553833747s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538220983 (195h40m58.553890598s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538621970 (195h40m58.554291573s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538674321 (195h40m58.554343932s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539132651 (195h40m58.554802257s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539598534 (195h40m58.55526814s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540272067 (195h40m58.555941683s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540830850 (195h40m58.556500455s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541282021 (195h40m58.556951628s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541844634 (195h40m58.557514243s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.542524800 (195h40m58.5581944s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:56.094849377 (195h40m59.110518971s) [upload=187967]: Next returned nil segment Mar 18 19:51:56.094868131 (195h40m59.110537734s) [upload=187967]: Waiting for error group managing segments... Mar 18 19:51:56.245291856 (195h40m59.260961473s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.255753691 (195h40m59.271423291s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.265834815 (195h40m59.281504424s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.279634521 (195h40m59.29530413s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.331060667 (195h40m59.346730277s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.408985715 (195h40m59.42465532s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.567711563 (195h40m59.583381165s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.648588128 (195h40m59.664257736s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.764700170 (195h40m59.780369777s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.769699005 (195h40m59.785368613s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.798647149 (195h40m59.814316756s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.951780070 (195h40m59.967449674s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.967179836 (195h40m59.982849436s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:52:51.120145929 (195h41m54.135815539s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:145.79.227.43:4172): protocol: rpc: dial tcp 145.79.227.43:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.120853163 (195h41m54.136522772s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1DrABVmM8aDtLyU41WcusdYx8JhnPvr9jgVWTcqduXs7CWBd7v, address:145.79.227.46:4610): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:52:51.120935993 (195h41m54.136605599s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: rpc: dial tcp 145.79.227.5:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121101831 (195h41m54.136771436s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12NkWHX2ojDAfX4CHoHSDE5mrLczUHwWQwm9rKPU7NUUB2GJgqY, address:145.79.227.9:4397): protocol: rpc: dial tcp 145.79.227.9:4397: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121229659 (195h41m54.13689927s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK, address:145.79.227.49:4367): protocol: rpc: dial tcp 145.79.227.49:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121281013 (195h41m54.136950616s) [seg_pos={0 0}, upload=187967]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:52:51.239544869 (195h41m54.25521447s) [upload=187967]: Done uploading segments... Mar 18 19:52:51.239583133 (195h41m54.255252738s) [upload=187967]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.353470358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 18.544496486s trace: https://jaeger.dev.storj.io/trace/308392e7c30c2aef FAIL (2026-03-18 19:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 19:52:53.807043673 (195h41m56.8227133s) [upload=187985]: Starting upload Mar 18 19:52:53.807204416 (195h41m56.822874017s) [upload=187985]: Done starting upload Mar 18 19:52:53.807276033 (195h41m56.822945623s) [upload=187985]: Uploading segments... Mar 18 19:52:53.807668631 (195h41m56.823338231s) [upload=187985]: Got next segment. Inline: false Mar 18 19:52:53.807720745 (195h41m56.823390345s) [upload=187985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:52:53.807725331 (195h41m56.823394935s) [upload=187985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:52:57.879544017 (195h42m0.895213607s) [seg_pos={0 0}, upload=187985]: Begin upload segment... Mar 18 19:52:57.879611055 (195h42m0.895280647s) [seg_pos={0 0}, upload=187985]: Done begin upload segment. Mar 18 19:52:57.879657998 (195h42m0.895327588s) [noise=true , address=45.91.116.7:4011 , piece_id=F2BLOLQHU76XFK57VLBSCCFN4RPCPWMOG5T434SFL532NJFX6QPA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.880051034 (195h42m0.895720644s) [noise=true , address=45.91.116.10:4088 , piece_id=WXYALSZMRFH3DFQRQBUSH7JV3KKH4XWMXW7HTUI3HAONDTOJ7QUA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.880991708 (195h42m0.896661317s) [noise=true , address=145.79.227.35:4482 , piece_id=JP3GUEAVDB3NVCCM2GPDTHCTSABRS7RTWCS4BXC5J7VGED7EKPHQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.881486954 (195h42m0.897156563s) [noise=true , address=79.127.219.41:4926 , piece_id=BJ3PCNQ54BWAOTW4Y3OGX5DBWULKTEMDEWZLFM635MYJAE2ALZLA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.881735859 (195h42m0.8974055s) [noise=true , address=45.91.116.4:4941 , piece_id=IC2WH22Y4FHVEUPNSQJVT7AGGWSH4KZLH64RXYXZJWR4DPXSS45A, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882122739 (195h42m0.897792342s) [noise=true , address=145.79.227.23:4551 , piece_id=DKVE7CXU3GUPMFMHNHLCUJFORWD2AU6TD64ODFOMEYH3AMINN33Q, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882518990 (195h42m0.898188602s) [noise=true , address=45.91.116.5:4300 , piece_id=F5H2ZOZQBZEVYGZZ4BY2RO2RUYU5ST3CHEDEJQ3MFWD4BFICBEDQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882542895 (195h42m0.898212498s) [noise=true , address=77.90.31.10:4517 , piece_id=JPKMQB733QDSXU4ZOGVYP7TMGQU7ZADQCPFBGWQ7Y26B2ARMV7IA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.883547656 (195h42m0.899217264s) [noise=true , address=77.90.31.13:4625 , piece_id=N6ZOOOEVQVPKNX3KQODEMVO5GNZCRKLWWJNEVL5Y33OJYF4P7EBQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.883682057 (195h42m0.899351662s) [noise=true , address=145.79.227.12:4771 , piece_id=IP5JDREK7XAXICS3VUDBFGG5TCJKNGUWX7PP6F4VDUO5TCMHRDUA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.884207666 (195h42m0.899877266s) [noise=true , address=145.79.227.9:4370 , piece_id=OW4WDQVK37ZLZVY24L765MA4PDMSCR5METIJUE3OB42TOSX66KPQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.884621968 (195h42m0.900291585s) [noise=true , address=77.90.31.16:4736 , piece_id=EZTBARQ3E5UKTFR44RAH4VY3D5LM2CS22BG4C47KCIGPF6FCRTCA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.885747622 (195h42m0.901417225s) [noise=true , address=77.90.31.15:4700 , piece_id=VWWCZ2TWGA7VHT2AHUDAJBUO7TZ5IQZYOHINNKB2O6N5EAFLE3FA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.886054881 (195h42m0.90172449s) [noise=true , address=145.79.227.46:4603 , piece_id=QDDZBHYLMIYIABWE5KBKABOFFTLAO7FASXZOOAA47453G6ZXDEDQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.886953815 (195h42m0.902623424s) [noise=true , address=145.79.227.18:4011 , piece_id=FUB6OMFHKMJ3P4GNURD7GPN24HKOUO7E2EXYPSAFUQI763JGU5MQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.887646187 (195h42m0.903315798s) [noise=true , address=145.79.227.21:4478 , piece_id=GCRM55WDWRLDDQPW5JUQ4XXJRQKEKIDBK4ACPDORZRVMFATW5LSA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:58.485511018 (195h42m1.501180619s) [upload=187985]: Next returned nil segment Mar 18 19:52:58.485527721 (195h42m1.501197328s) [upload=187985]: Waiting for error group managing segments... Mar 18 19:52:58.620445775 (195h42m1.636115378s) [noise=true , address=77.90.31.15:4700 , piece_id=VWWCZ2TWGA7VHT2AHUDAJBUO7TZ5IQZYOHINNKB2O6N5EAFLE3FA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.627168027 (195h42m1.642837637s) [noise=true , address=45.91.116.4:4941 , piece_id=IC2WH22Y4FHVEUPNSQJVT7AGGWSH4KZLH64RXYXZJWR4DPXSS45A, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.631350617 (195h42m1.647020218s) [noise=true , address=77.90.31.16:4736 , piece_id=EZTBARQ3E5UKTFR44RAH4VY3D5LM2CS22BG4C47KCIGPF6FCRTCA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.634350072 (195h42m1.650019672s) [noise=true , address=79.127.219.41:4926 , piece_id=BJ3PCNQ54BWAOTW4Y3OGX5DBWULKTEMDEWZLFM635MYJAE2ALZLA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.639838114 (195h42m1.655507715s) [noise=true , address=45.91.116.10:4088 , piece_id=WXYALSZMRFH3DFQRQBUSH7JV3KKH4XWMXW7HTUI3HAONDTOJ7QUA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.642299884 (195h42m1.657969484s) [noise=true , address=45.91.116.5:4300 , piece_id=F5H2ZOZQBZEVYGZZ4BY2RO2RUYU5ST3CHEDEJQ3MFWD4BFICBEDQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.643583570 (195h42m1.65925317s) [noise=true , address=45.91.116.7:4011 , piece_id=F2BLOLQHU76XFK57VLBSCCFN4RPCPWMOG5T434SFL532NJFX6QPA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.644248240 (195h42m1.659917844s) [noise=true , address=77.90.31.10:4517 , piece_id=JPKMQB733QDSXU4ZOGVYP7TMGQU7ZADQCPFBGWQ7Y26B2ARMV7IA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.644949824 (195h42m1.660619425s) [noise=true , address=77.90.31.13:4625 , piece_id=N6ZOOOEVQVPKNX3KQODEMVO5GNZCRKLWWJNEVL5Y33OJYF4P7EBQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:53:23.507533028 (195h42m26.523202635s) [noise=true , address=145.79.227.21:4478 , piece_id=GCRM55WDWRLDDQPW5JUQ4XXJRQKEKIDBK4ACPDORZRVMFATW5LSA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:145.79.227.21:4478): protocol: rpc: dial tcp 145.79.227.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507609476 (195h42m26.523279082s) [noise=true , address=145.79.227.46:4603 , piece_id=QDDZBHYLMIYIABWE5KBKABOFFTLAO7FASXZOOAA47453G6ZXDEDQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507791537 (195h42m26.52346114s) [noise=true , address=145.79.227.12:4771 , piece_id=IP5JDREK7XAXICS3VUDBFGG5TCJKNGUWX7PP6F4VDUO5TCMHRDUA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:145.79.227.12:4771): protocol: rpc: dial tcp 145.79.227.12:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507844065 (195h42m26.523513674s) [noise=true , address=145.79.227.23:4551 , piece_id=DKVE7CXU3GUPMFMHNHLCUJFORWD2AU6TD64ODFOMEYH3AMINN33Q, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12vR5WRMGoYyzYDZzwELFw6kLG6vtKsUeFM9kUhgZDLGKB6vcXs, address:145.79.227.23:4551): protocol: rpc: dial tcp 145.79.227.23:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507968973 (195h42m26.523638579s) [noise=true , address=145.79.227.35:4482 , piece_id=JP3GUEAVDB3NVCCM2GPDTHCTSABRS7RTWCS4BXC5J7VGED7EKPHQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.508179982 (195h42m26.523849591s) [noise=true , address=145.79.227.9:4370 , piece_id=OW4WDQVK37ZLZVY24L765MA4PDMSCR5METIJUE3OB42TOSX66KPQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm, address:145.79.227.9:4370): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:53:23.508461286 (195h42m26.5241309s) [noise=true , address=145.79.227.18:4011 , piece_id=FUB6OMFHKMJ3P4GNURD7GPN24HKOUO7E2EXYPSAFUQI763JGU5MQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:145.79.227.18:4011): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:53:23.508499349 (195h42m26.524168944s) [seg_pos={0 0}, upload=187985]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 19:53:23.620162279 (195h42m26.635831867s) [upload=187985]: Done uploading segments... Mar 18 19:53:23.620199162 (195h42m26.635868761s) [upload=187985]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 18.544528418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.................................XXXXXXXX....XX.XXXXXXXXXXXX] duration: 5.001044897s trace: https://jaeger.dev.storj.io/trace/7f66679376d451ea FAIL (2026-03-18 19:53:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001093077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.116223415s trace: https://jaeger.dev.storj.io/trace/308392e7c30c2aef FAIL (2026-03-18 19:53:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 19:52:53.807043673 (195h41m56.8227133s) [upload=187985]: Starting upload Mar 18 19:52:53.807204416 (195h41m56.822874017s) [upload=187985]: Done starting upload Mar 18 19:52:53.807276033 (195h41m56.822945623s) [upload=187985]: Uploading segments... Mar 18 19:52:53.807668631 (195h41m56.823338231s) [upload=187985]: Got next segment. Inline: false Mar 18 19:52:53.807720745 (195h41m56.823390345s) [upload=187985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:52:53.807725331 (195h41m56.823394935s) [upload=187985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:52:57.879544017 (195h42m0.895213607s) [seg_pos={0 0}, upload=187985]: Begin upload segment... Mar 18 19:52:57.879611055 (195h42m0.895280647s) [seg_pos={0 0}, upload=187985]: Done begin upload segment. Mar 18 19:52:57.879657998 (195h42m0.895327588s) [noise=true , address=45.91.116.7:4011 , piece_id=F2BLOLQHU76XFK57VLBSCCFN4RPCPWMOG5T434SFL532NJFX6QPA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.880051034 (195h42m0.895720644s) [noise=true , address=45.91.116.10:4088 , piece_id=WXYALSZMRFH3DFQRQBUSH7JV3KKH4XWMXW7HTUI3HAONDTOJ7QUA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.880991708 (195h42m0.896661317s) [noise=true , address=145.79.227.35:4482 , piece_id=JP3GUEAVDB3NVCCM2GPDTHCTSABRS7RTWCS4BXC5J7VGED7EKPHQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.881486954 (195h42m0.897156563s) [noise=true , address=79.127.219.41:4926 , piece_id=BJ3PCNQ54BWAOTW4Y3OGX5DBWULKTEMDEWZLFM635MYJAE2ALZLA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.881735859 (195h42m0.8974055s) [noise=true , address=45.91.116.4:4941 , piece_id=IC2WH22Y4FHVEUPNSQJVT7AGGWSH4KZLH64RXYXZJWR4DPXSS45A, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882122739 (195h42m0.897792342s) [noise=true , address=145.79.227.23:4551 , piece_id=DKVE7CXU3GUPMFMHNHLCUJFORWD2AU6TD64ODFOMEYH3AMINN33Q, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882518990 (195h42m0.898188602s) [noise=true , address=45.91.116.5:4300 , piece_id=F5H2ZOZQBZEVYGZZ4BY2RO2RUYU5ST3CHEDEJQ3MFWD4BFICBEDQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.882542895 (195h42m0.898212498s) [noise=true , address=77.90.31.10:4517 , piece_id=JPKMQB733QDSXU4ZOGVYP7TMGQU7ZADQCPFBGWQ7Y26B2ARMV7IA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.883547656 (195h42m0.899217264s) [noise=true , address=77.90.31.13:4625 , piece_id=N6ZOOOEVQVPKNX3KQODEMVO5GNZCRKLWWJNEVL5Y33OJYF4P7EBQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.883682057 (195h42m0.899351662s) [noise=true , address=145.79.227.12:4771 , piece_id=IP5JDREK7XAXICS3VUDBFGG5TCJKNGUWX7PP6F4VDUO5TCMHRDUA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.884207666 (195h42m0.899877266s) [noise=true , address=145.79.227.9:4370 , piece_id=OW4WDQVK37ZLZVY24L765MA4PDMSCR5METIJUE3OB42TOSX66KPQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.884621968 (195h42m0.900291585s) [noise=true , address=77.90.31.16:4736 , piece_id=EZTBARQ3E5UKTFR44RAH4VY3D5LM2CS22BG4C47KCIGPF6FCRTCA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.885747622 (195h42m0.901417225s) [noise=true , address=77.90.31.15:4700 , piece_id=VWWCZ2TWGA7VHT2AHUDAJBUO7TZ5IQZYOHINNKB2O6N5EAFLE3FA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.886054881 (195h42m0.90172449s) [noise=true , address=145.79.227.46:4603 , piece_id=QDDZBHYLMIYIABWE5KBKABOFFTLAO7FASXZOOAA47453G6ZXDEDQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.886953815 (195h42m0.902623424s) [noise=true , address=145.79.227.18:4011 , piece_id=FUB6OMFHKMJ3P4GNURD7GPN24HKOUO7E2EXYPSAFUQI763JGU5MQ, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:57.887646187 (195h42m0.903315798s) [noise=true , address=145.79.227.21:4478 , piece_id=GCRM55WDWRLDDQPW5JUQ4XXJRQKEKIDBK4ACPDORZRVMFATW5LSA, seg_pos={0 0}, upload=187985]: Uploading piece... Mar 18 19:52:58.485511018 (195h42m1.501180619s) [upload=187985]: Next returned nil segment Mar 18 19:52:58.485527721 (195h42m1.501197328s) [upload=187985]: Waiting for error group managing segments... Mar 18 19:52:58.620445775 (195h42m1.636115378s) [noise=true , address=77.90.31.15:4700 , piece_id=VWWCZ2TWGA7VHT2AHUDAJBUO7TZ5IQZYOHINNKB2O6N5EAFLE3FA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.627168027 (195h42m1.642837637s) [noise=true , address=45.91.116.4:4941 , piece_id=IC2WH22Y4FHVEUPNSQJVT7AGGWSH4KZLH64RXYXZJWR4DPXSS45A, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.631350617 (195h42m1.647020218s) [noise=true , address=77.90.31.16:4736 , piece_id=EZTBARQ3E5UKTFR44RAH4VY3D5LM2CS22BG4C47KCIGPF6FCRTCA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.634350072 (195h42m1.650019672s) [noise=true , address=79.127.219.41:4926 , piece_id=BJ3PCNQ54BWAOTW4Y3OGX5DBWULKTEMDEWZLFM635MYJAE2ALZLA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.639838114 (195h42m1.655507715s) [noise=true , address=45.91.116.10:4088 , piece_id=WXYALSZMRFH3DFQRQBUSH7JV3KKH4XWMXW7HTUI3HAONDTOJ7QUA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.642299884 (195h42m1.657969484s) [noise=true , address=45.91.116.5:4300 , piece_id=F5H2ZOZQBZEVYGZZ4BY2RO2RUYU5ST3CHEDEJQ3MFWD4BFICBEDQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.643583570 (195h42m1.65925317s) [noise=true , address=45.91.116.7:4011 , piece_id=F2BLOLQHU76XFK57VLBSCCFN4RPCPWMOG5T434SFL532NJFX6QPA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.644248240 (195h42m1.659917844s) [noise=true , address=77.90.31.10:4517 , piece_id=JPKMQB733QDSXU4ZOGVYP7TMGQU7ZADQCPFBGWQ7Y26B2ARMV7IA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:52:58.644949824 (195h42m1.660619425s) [noise=true , address=77.90.31.13:4625 , piece_id=N6ZOOOEVQVPKNX3KQODEMVO5GNZCRKLWWJNEVL5Y33OJYF4P7EBQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: <nil> Mar 18 19:53:23.507533028 (195h42m26.523202635s) [noise=true , address=145.79.227.21:4478 , piece_id=GCRM55WDWRLDDQPW5JUQ4XXJRQKEKIDBK4ACPDORZRVMFATW5LSA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:145.79.227.21:4478): protocol: rpc: dial tcp 145.79.227.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507609476 (195h42m26.523279082s) [noise=true , address=145.79.227.46:4603 , piece_id=QDDZBHYLMIYIABWE5KBKABOFFTLAO7FASXZOOAA47453G6ZXDEDQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507791537 (195h42m26.52346114s) [noise=true , address=145.79.227.12:4771 , piece_id=IP5JDREK7XAXICS3VUDBFGG5TCJKNGUWX7PP6F4VDUO5TCMHRDUA, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:145.79.227.12:4771): protocol: rpc: dial tcp 145.79.227.12:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507844065 (195h42m26.523513674s) [noise=true , address=145.79.227.23:4551 , piece_id=DKVE7CXU3GUPMFMHNHLCUJFORWD2AU6TD64ODFOMEYH3AMINN33Q, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12vR5WRMGoYyzYDZzwELFw6kLG6vtKsUeFM9kUhgZDLGKB6vcXs, address:145.79.227.23:4551): protocol: rpc: dial tcp 145.79.227.23:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.507968973 (195h42m26.523638579s) [noise=true , address=145.79.227.35:4482 , piece_id=JP3GUEAVDB3NVCCM2GPDTHCTSABRS7RTWCS4BXC5J7VGED7EKPHQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:145.79.227.35:4482): protocol: rpc: dial tcp 145.79.227.35:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:53:23.508179982 (195h42m26.523849591s) [noise=true , address=145.79.227.9:4370 , piece_id=OW4WDQVK37ZLZVY24L765MA4PDMSCR5METIJUE3OB42TOSX66KPQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12SXZN7WJvcHys9BRd4Lq8NAZ3MSgHf9ZVv7SgaiJNRosdmFVpm, address:145.79.227.9:4370): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:53:23.508461286 (195h42m26.5241309s) [noise=true , address=145.79.227.18:4011 , piece_id=FUB6OMFHKMJ3P4GNURD7GPN24HKOUO7E2EXYPSAFUQI763JGU5MQ, seg_pos={0 0}, upload=187985]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:145.79.227.18:4011): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:53:23.508499349 (195h42m26.524168944s) [seg_pos={0 0}, upload=187985]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 19:53:23.620162279 (195h42m26.635831867s) [upload=187985]: Done uploading segments... Mar 18 19:53:23.620199162 (195h42m26.635868761s) [upload=187985]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.116253134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................X..X...X..X..X........X.] duration: 23.494834264s trace: https://jaeger.dev.storj.io/trace/20c3e3002cfafbf1 FAIL (2026-03-18 19:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== tarting upload Mar 18 19:51:51.422439741 (195h40m54.43810934s) [upload=187967]: Uploading segments... Mar 18 19:51:51.422756046 (195h40m54.438425655s) [upload=187967]: Got next segment. Inline: false Mar 18 19:51:51.422805418 (195h40m54.438475023s) [upload=187967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:51:51.422814967 (195h40m54.438484569s) [upload=187967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:51:55.535636400 (195h40m58.551306s) [seg_pos={0 0}, upload=187967]: Begin upload segment... Mar 18 19:51:55.535747846 (195h40m58.551417436s) [seg_pos={0 0}, upload=187967]: Done begin upload segment. Mar 18 19:51:55.535781095 (195h40m58.551450701s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.535809704 (195h40m58.551479313s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536014863 (195h40m58.551684463s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536516173 (195h40m58.552185775s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536952649 (195h40m58.55262225s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537353909 (195h40m58.553023508s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537775660 (195h40m58.55344528s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538164137 (195h40m58.553833747s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538220983 (195h40m58.553890598s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538621970 (195h40m58.554291573s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538674321 (195h40m58.554343932s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539132651 (195h40m58.554802257s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539598534 (195h40m58.55526814s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540272067 (195h40m58.555941683s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540830850 (195h40m58.556500455s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541282021 (195h40m58.556951628s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541844634 (195h40m58.557514243s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.542524800 (195h40m58.5581944s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:56.094849377 (195h40m59.110518971s) [upload=187967]: Next returned nil segment Mar 18 19:51:56.094868131 (195h40m59.110537734s) [upload=187967]: Waiting for error group managing segments... Mar 18 19:51:56.245291856 (195h40m59.260961473s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.255753691 (195h40m59.271423291s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.265834815 (195h40m59.281504424s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.279634521 (195h40m59.29530413s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.331060667 (195h40m59.346730277s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.408985715 (195h40m59.42465532s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.567711563 (195h40m59.583381165s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.648588128 (195h40m59.664257736s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.764700170 (195h40m59.780369777s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.769699005 (195h40m59.785368613s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.798647149 (195h40m59.814316756s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.951780070 (195h40m59.967449674s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.967179836 (195h40m59.982849436s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:52:51.120145929 (195h41m54.135815539s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:145.79.227.43:4172): protocol: rpc: dial tcp 145.79.227.43:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.120853163 (195h41m54.136522772s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1DrABVmM8aDtLyU41WcusdYx8JhnPvr9jgVWTcqduXs7CWBd7v, address:145.79.227.46:4610): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:52:51.120935993 (195h41m54.136605599s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: rpc: dial tcp 145.79.227.5:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121101831 (195h41m54.136771436s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12NkWHX2ojDAfX4CHoHSDE5mrLczUHwWQwm9rKPU7NUUB2GJgqY, address:145.79.227.9:4397): protocol: rpc: dial tcp 145.79.227.9:4397: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121229659 (195h41m54.13689927s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK, address:145.79.227.49:4367): protocol: rpc: dial tcp 145.79.227.49:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121281013 (195h41m54.136950616s) [seg_pos={0 0}, upload=187967]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:52:51.239544869 (195h41m54.25521447s) [upload=187967]: Done uploading segments... Mar 18 19:52:51.239583133 (195h41m54.255252738s) [upload=187967]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.494852557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXX.XXX...............] duration: 19.868493768s trace: https://jaeger.dev.storj.io/trace/4d6a8ab787f335c0 FAIL (2026-03-18 19:53:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.868520238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..................................XXXXXXXX....XX.XXXXXXXXXXX] duration: 5.001119192s trace: https://jaeger.dev.storj.io/trace/4a3e7e4a03df3288 FAIL (2026-03-18 19:52:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001170465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X..X...X..X..X........X] duration: 1m0.121283759s trace: https://jaeger.dev.storj.io/trace/20c3e3002cfafbf1 FAIL (2026-03-18 19:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== tarting upload Mar 18 19:51:51.422439741 (195h40m54.43810934s) [upload=187967]: Uploading segments... Mar 18 19:51:51.422756046 (195h40m54.438425655s) [upload=187967]: Got next segment. Inline: false Mar 18 19:51:51.422805418 (195h40m54.438475023s) [upload=187967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:51:51.422814967 (195h40m54.438484569s) [upload=187967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:51:55.535636400 (195h40m58.551306s) [seg_pos={0 0}, upload=187967]: Begin upload segment... Mar 18 19:51:55.535747846 (195h40m58.551417436s) [seg_pos={0 0}, upload=187967]: Done begin upload segment. Mar 18 19:51:55.535781095 (195h40m58.551450701s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.535809704 (195h40m58.551479313s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536014863 (195h40m58.551684463s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536516173 (195h40m58.552185775s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.536952649 (195h40m58.55262225s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537353909 (195h40m58.553023508s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.537775660 (195h40m58.55344528s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538164137 (195h40m58.553833747s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538220983 (195h40m58.553890598s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538621970 (195h40m58.554291573s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.538674321 (195h40m58.554343932s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539132651 (195h40m58.554802257s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.539598534 (195h40m58.55526814s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540272067 (195h40m58.555941683s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.540830850 (195h40m58.556500455s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541282021 (195h40m58.556951628s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.541844634 (195h40m58.557514243s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:55.542524800 (195h40m58.5581944s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Uploading piece... Mar 18 19:51:56.094849377 (195h40m59.110518971s) [upload=187967]: Next returned nil segment Mar 18 19:51:56.094868131 (195h40m59.110537734s) [upload=187967]: Waiting for error group managing segments... Mar 18 19:51:56.245291856 (195h40m59.260961473s) [noise=true , address=45.91.116.11:4124 , piece_id=EL4ZFLY2AWQOQRBE2LD5W3Z6YQHCTGAQMKKCTR6EB6TR3GTNJQVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.255753691 (195h40m59.271423291s) [noise=true , address=79.127.219.45:4877 , piece_id=YOAJAOYSP3ZTRKMV2GCS43BXMW3AAEB6DQMPGU4HM73GHKNS2T5Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.265834815 (195h40m59.281504424s) [noise=true , address=45.91.116.6:4982 , piece_id=NGNYTWWKEXCQUOX3QZHFOJDLX4UJI6IEFYG5LJCSVPCXYEGKNTRA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.279634521 (195h40m59.29530413s) [noise=true , address=77.90.31.21:4886 , piece_id=4D5CKTEDEHLNFPAARTNIW5UFPBW4L6IOJQRHPC56OKRR5GSLBHMA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.331060667 (195h40m59.346730277s) [noise=true , address=77.90.31.19:4852 , piece_id=5OYDBHT25G54IYCCRUOCFXI6ULU7ZXJDQU5MURFXAM4RYS7HTH3Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.408985715 (195h40m59.42465532s) [noise=true , address=45.91.116.12:4247 , piece_id=VBDYLYX45GU3NXBOIHPPEAFN7HAI2H7VNRXIDE5FH6H4AXWTFVLA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.567711563 (195h40m59.583381165s) [noise=true , address=167.17.72.68:4662 , piece_id=LOHPUFTKSSLVHIYV6ZGDWBEWQAFWYU5GCMMEFJKFN74JOWUTACYA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.648588128 (195h40m59.664257736s) [noise=true , address=167.17.72.71:4797 , piece_id=H2SZO7IUDRSTGAN65GBRE2OL3Y5W5LEQUCNA2HDL7PLI6PLNZEVQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.764700170 (195h40m59.780369777s) [noise=true , address=193.149.164.71:4286 , piece_id=MM7RGRFUPT3XX4IVVDXITG42NEXML6VDUQHEKRAYDGKDAE7OHAWQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.769699005 (195h40m59.785368613s) [noise=true , address=193.149.164.70:4126 , piece_id=FH7LYTZ2YHGEDCZI2DUFU5W6T7AC5IZAKZUN5U3O4TZSYTLYGG6Q, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.798647149 (195h40m59.814316756s) [noise=true , address=193.149.164.67:4023 , piece_id=YEMVTQZDFYJ6LIHSCTMSFTUPVT24OR4QQZQJC4BH7KKHLOIAL4ZA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.951780070 (195h40m59.967449674s) [noise=true , address=79.127.205.234:4193 , piece_id=MMTAHNQRFWE5CUWZHWP2MMHSJEIPVS74BCINF5UJOWTZTF7N36KA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:51:56.967179836 (195h40m59.982849436s) [noise=true , address=79.127.205.233:4162 , piece_id=6PEAFTY3ZEM4GS22XU67PCZZVQZDTCWPTBT3CXDRWEM2ADUDIYLQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: <nil> Mar 18 19:52:51.120145929 (195h41m54.135815539s) [noise=true , address=145.79.227.43:4172 , piece_id=LCKKZPY2FOD2HZRERXRODYS3U3DDXJRQOT6RFULO6R53PAJFA2QA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:145.79.227.43:4172): protocol: rpc: dial tcp 145.79.227.43:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.120853163 (195h41m54.136522772s) [noise=true , address=145.79.227.46:4610 , piece_id=BCM5GCSTS3BLPTIQDXRMCB3BHHKCBORWSMSKU5EAGD7VL4YJHMOA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:1DrABVmM8aDtLyU41WcusdYx8JhnPvr9jgVWTcqduXs7CWBd7v, address:145.79.227.46:4610): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:52:51.120935993 (195h41m54.136605599s) [noise=true , address=145.79.227.5:4238 , piece_id=WLOLBQDHECKBFBOJFCGNQVBGI5V3KI3GSQNDZ42LMDDJJRHLDEUQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: rpc: dial tcp 145.79.227.5:4238: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121101831 (195h41m54.136771436s) [noise=true , address=145.79.227.9:4397 , piece_id=VFT27PPV7BCDPZOHA6A4D5OIZY37CHQN6KICE2CXJCZ6XUKYKBHA, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12NkWHX2ojDAfX4CHoHSDE5mrLczUHwWQwm9rKPU7NUUB2GJgqY, address:145.79.227.9:4397): protocol: rpc: dial tcp 145.79.227.9:4397: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121229659 (195h41m54.13689927s) [noise=true , address=145.79.227.49:4367 , piece_id=VSSUVOT2MIDF2YVCYWK52V5GNL7MM4PXPXKGKWM7PO4N5EZFIWYQ, seg_pos={0 0}, upload=187967]: Done uploading piece. err: ecclient: upload failed (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK, address:145.79.227.49:4367): protocol: rpc: dial tcp 145.79.227.49:4367: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:51.121281013 (195h41m54.136950616s) [seg_pos={0 0}, upload=187967]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 18 19:52:51.239544869 (195h41m54.25521447s) [upload=187967]: Done uploading segments... Mar 18 19:52:51.239583133 (195h41m54.255252738s) [upload=187967]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.121308593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.112794351s trace: https://jaeger.dev.storj.io/trace/738aaffec3c48088 FAIL (2026-03-18 19:52:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Mar 18 19:52:00.866259777 (195h41m3.881929379s) [upload=187971]: Starting upload Mar 18 19:52:00.866371836 (195h41m3.882041445s) [upload=187971]: Done starting upload Mar 18 19:52:00.866419568 (195h41m3.882089166s) [upload=187971]: Uploading segments... Mar 18 19:52:00.866737929 (195h41m3.88240753s) [upload=187971]: Got next segment. Inline: false Mar 18 19:52:00.866812815 (195h41m3.882482425s) [upload=187971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:52:00.866819615 (195h41m3.882489226s) [upload=187971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:52:04.842558529 (195h41m7.858228142s) [seg_pos={0 0}, upload=187971]: Begin upload segment... Mar 18 19:52:04.842630288 (195h41m7.858299901s) [seg_pos={0 0}, upload=187971]: Done begin upload segment. Mar 18 19:52:04.842673813 (195h41m7.858343428s) [noise=true , address=145.79.227.23:4548 , piece_id=VGZUJ6OZRBXHY6V5WC6P4464CMAAH6AOPX7HIDSH3AKED2ECBO6A, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.846382695 (195h41m7.862052309s) [noise=true , address=45.91.116.12:4226 , piece_id=ZDIQ5DYDUCAF772DA45RAQO22ZSNRZVRMNWNYKELBMA6FPMCSA6A, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.846422210 (195h41m7.862091813s) [noise=true , address=145.79.227.31:4801 , piece_id=KSCEJPY4IQYQCEWVMQ2R7QG26PPBOEALOI2SB6W5LIQ5HHZH6JNA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.846511867 (195h41m7.862181456s) [noise=true , address=77.90.31.17:4769 , piece_id=RFPDETHPCSQPC4NMPZV7OETILZUPS5AZEKKQVK6LNEFDY2GLFGGQ, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.847196224 (195h41m7.862865838s) [noise=true , address=45.91.116.14:4196 , piece_id=BHOL7SLFALZOPAV67ZSHZYEWU7K47E3IRT56APBDRJ5AQ6JUJB4A, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.847573967 (195h41m7.863243582s) [noise=true , address=145.79.227.49:4338 , piece_id=TTEFX4E4WOYAX6JDJMVEZYGJAKRUXF7OTD2IB4RWPTC4MPAKUW4A, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.856157766 (195h41m7.87182738s) [noise=true , address=45.91.116.9:4296 , piece_id=DWKGHCR7VFK7BVBJYONGTJLBSKG5ELAZLZ5B53UC2ESCLDI5EIGQ, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.859170832 (195h41m7.874840445s) [noise=true , address=45.91.116.7:4009 , piece_id=LLP6VTRQW7YCMG7ERVS6L73XXPA43HDSJ3GLXZ45ME2BFGDTNSSA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.860762329 (195h41m7.876431942s) [noise=true , address=145.79.227.5:4227 , piece_id=HLZSF5TWEATKBQ52MU4EHZ4SLF4NH74TPUNBVQUBKZSJ2UJUS5HA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.862643984 (195h41m7.878313594s) [noise=true , address=77.90.31.13:4627 , piece_id=5YAWYU745EQF43C6DFYOQQMP76TZ4DBESLLWH72RUPQ6OZEMRNKA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.863691080 (195h41m7.879360669s) [noise=true , address=79.127.219.46:4894 , piece_id=NE4QUCVEARU3HR42J5JP5MGZSUXR327OETD72ZCLOXFHJQECQN3Q, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.865330561 (195h41m7.881000169s) [noise=true , address=145.79.227.9:4369 , piece_id=JPIEEU6D45KKHAQIH35FNPUB2NF7U6VO5E26PG5LEQNZIKIC7HYQ, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.866512063 (195h41m7.882181755s) [noise=true , address=77.90.31.16:4736 , piece_id=LRPCJJJNY6EDEUEEZIPKF6OKK3G7RQ5AXEFTNVQ2V62KQKM7XRTQ, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.867867656 (195h41m7.883537266s) [noise=true , address=45.91.116.13:4156 , piece_id=BK56XTWODHYKTAGDEKA74Y26CH6CYDFVTAEEV3NCI5BCC4JEVHPA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.869307653 (195h41m7.884977267s) [noise=true , address=77.90.31.14:4665 , piece_id=ZJSDFJNX72NTGL2SJF2HCXDNMEP7HSS7HR7AAY655QL2EPTCPEIQ, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:04.870468619 (195h41m7.886138234s) [noise=true , address=145.79.227.40:4052 , piece_id=B6TSWEA2NKI24AW3QTSZSJOEBES7XYWHEGFANCGDLH72FM4RG2BA, seg_pos={0 0}, upload=187971]: Uploading piece... Mar 18 19:52:05.487223198 (195h41m8.502892809s) [upload=187971]: Next returned nil segment Mar 18 19:52:05.487242207 (195h41m8.50291182s) [upload=187971]: Waiting for error group managing segments... Mar 18 19:52:05.609369354 (195h41m8.625038962s) [noise=true , address=77.90.31.13:4627 , piece_id=5YAWYU745EQF43C6DFYOQQMP76TZ4DBESLLWH72RUPQ6OZEMRNKA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.615580164 (195h41m8.631249773s) [noise=true , address=45.91.116.13:4156 , piece_id=BK56XTWODHYKTAGDEKA74Y26CH6CYDFVTAEEV3NCI5BCC4JEVHPA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.616413574 (195h41m8.63208318s) [noise=true , address=77.90.31.14:4665 , piece_id=ZJSDFJNX72NTGL2SJF2HCXDNMEP7HSS7HR7AAY655QL2EPTCPEIQ, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.619086527 (195h41m8.63475613s) [noise=true , address=45.91.116.9:4296 , piece_id=DWKGHCR7VFK7BVBJYONGTJLBSKG5ELAZLZ5B53UC2ESCLDI5EIGQ, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.619106055 (195h41m8.634775673s) [noise=true , address=45.91.116.7:4009 , piece_id=LLP6VTRQW7YCMG7ERVS6L73XXPA43HDSJ3GLXZ45ME2BFGDTNSSA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.619689574 (195h41m8.635359185s) [noise=true , address=77.90.31.16:4736 , piece_id=LRPCJJJNY6EDEUEEZIPKF6OKK3G7RQ5AXEFTNVQ2V62KQKM7XRTQ, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.621248205 (195h41m8.6369178s) [noise=true , address=45.91.116.14:4196 , piece_id=BHOL7SLFALZOPAV67ZSHZYEWU7K47E3IRT56APBDRJ5AQ6JUJB4A, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.624471473 (195h41m8.640141075s) [noise=true , address=77.90.31.17:4769 , piece_id=RFPDETHPCSQPC4NMPZV7OETILZUPS5AZEKKQVK6LNEFDY2GLFGGQ, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.624802337 (195h41m8.640471947s) [noise=true , address=45.91.116.12:4226 , piece_id=ZDIQ5DYDUCAF772DA45RAQO22ZSNRZVRMNWNYKELBMA6FPMCSA6A, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:05.715140531 (195h41m8.730810137s) [noise=true , address=79.127.219.46:4894 , piece_id=NE4QUCVEARU3HR42J5JP5MGZSUXR327OETD72ZCLOXFHJQECQN3Q, seg_pos={0 0}, upload=187971]: Done uploading piece. err: <nil> Mar 18 19:52:30.564116269 (195h41m33.579785875s) [noise=true , address=145.79.227.23:4548 , piece_id=VGZUJ6OZRBXHY6V5WC6P4464CMAAH6AOPX7HIDSH3AKED2ECBO6A, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:19rx4nU8dUTtPXvMonzZSBkn5Z5cnBXrndvU7t8ewUmfYLBqid, address:145.79.227.23:4548): protocol: rpc: dial tcp 145.79.227.23:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564246574 (195h41m33.57991618s) [noise=true , address=145.79.227.31:4801 , piece_id=KSCEJPY4IQYQCEWVMQ2R7QG26PPBOEALOI2SB6W5LIQ5HHZH6JNA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: rpc: dial tcp 145.79.227.31:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564351657 (195h41m33.580021264s) [noise=true , address=145.79.227.5:4227 , piece_id=HLZSF5TWEATKBQ52MU4EHZ4SLF4NH74TPUNBVQUBKZSJ2UJUS5HA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564408942 (195h41m33.580078546s) [noise=true , address=145.79.227.40:4052 , piece_id=B6TSWEA2NKI24AW3QTSZSJOEBES7XYWHEGFANCGDLH72FM4RG2BA, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V, address:145.79.227.40:4052): protocol: rpc: dial tcp 145.79.227.40:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564482188 (195h41m33.580151792s) [noise=true , address=145.79.227.9:4369 , piece_id=JPIEEU6D45KKHAQIH35FNPUB2NF7U6VO5E26PG5LEQNZIKIC7HYQ, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: rpc: dial tcp 145.79.227.9:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564549539 (195h41m33.580219139s) [noise=true , address=145.79.227.49:4338 , piece_id=TTEFX4E4WOYAX6JDJMVEZYGJAKRUXF7OTD2IB4RWPTC4MPAKUW4A, seg_pos={0 0}, upload=187971]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:52:30.564578759 (195h41m33.58024835s) [seg_pos={0 0}, upload=187971]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Mar 18 19:52:30.674308063 (195h41m33.689977664s) [upload=187971]: Done uploading segments... Mar 18 19:52:30.674344945 (195h41m33.690014545s) [upload=187971]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.112895131s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...................................XXXXXXXX....XX.XXXXXXXXXX] duration: 5.001358404s trace: https://jaeger.dev.storj.io/trace/2cafc9c6b3ff2e93 FAIL (2026-03-18 19:52:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00138655s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXX.XXX..............] duration: 21.767169353s trace: https://jaeger.dev.storj.io/trace/4d6a8ab787f335c0 FAIL (2026-03-18 19:52:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.767341837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.117002654s trace: https://jaeger.dev.storj.io/trace/b11693ff26a0929 FAIL (2026-03-18 19:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 7393925 (195h40m17.043063531s) [seg_pos={0 0}, upload=187956]: Done begin upload segment. Mar 18 19:51:14.027466313 (195h40m17.043135924s) [noise=true , address=79.127.219.45:4854 , piece_id=QVHKP5MKEWB3S5SKVI7425UPHMYV4UKYKO4GN25DDIJL4DIS4JKQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.027480453 (195h40m17.043150063s) [noise=true , address=145.79.227.2:4119 , piece_id=NREBXP2UBDE4DO44LFUTKOISJXM2QFAKX7M65JCJCVDRYOZTOPIQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.028186354 (195h40m17.043855957s) [noise=true , address=145.79.227.25:4619 , piece_id=N35DZJCJNH6EHHOBFA2UQUTYWJ6DCY54OIDVIEYQZ3AILNKUIJEQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.028389122 (195h40m17.044058732s) [noise=true , address=145.79.227.5:4226 , piece_id=OYKQ23W5OPHZV3MHPDTUC2D4RUWOFY2AKO3YAGRYASUS2AYJOPJQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.028829420 (195h40m17.044499031s) [noise=true , address=145.79.227.51:4410 , piece_id=7IMSVXUH22BHYLVJV66J6PFKOKU23VMWQMLISIN67XUBMDZGLBMA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.029207270 (195h40m17.044876861s) [noise=true , address=145.79.227.13:4806 , piece_id=MG47RNSKRGHSMRHA7ZD4XV3ODZD72DSJHYU27XCOYOBDKAQWSVHA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.029212588 (195h40m17.044882188s) [noise=true , address=45.91.116.13:4155 , piece_id=UG6JO6WWSQQ772OW4LCNH674XATGGP3AEDZHP6ZHS7SFVRI67CWA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.029837078 (195h40m17.045506687s) [noise=true , address=77.90.31.10:4519 , piece_id=FHF524X773L3ZTFOGROTJCQGXLBWGVKLSTHZDGLGBUJQAEHDWZ5A, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.029974245 (195h40m17.045643854s) [noise=true , address=145.79.227.3:4153 , piece_id=EWO6MGOBTUNM7LRDEBQKMRNAO2EMOMTSJ7WKXMUKWDUDDN2EIZRA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.030338092 (195h40m17.046007701s) [noise=true , address=145.79.227.12:4771 , piece_id=EPSFCRGJC374HJBYCXJEHW3JKD6BUNURQ466A5WOUQW2DRHA2MYA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.030543093 (195h40m17.046212704s) [noise=true , address=45.91.116.12:4260 , piece_id=3TKSZPWPYFJFHUKS3UAY7BSVLNYHE5JLVIYF2QRFGAAWE52YUMPQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.030825343 (195h40m17.04649495s) [noise=true , address=145.79.227.44:4198 , piece_id=6DZALCMGRZTL5WLUSI6RFTPNS3MPI4PW35K4JOSZENYMECLM4QTA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.031125897 (195h40m17.046795501s) [noise=true , address=145.79.227.45:4233 , piece_id=KC7IHYX32KX3AECFS2HUKZMU766BWY3MPGLNW5H5IUEHJ7OJZDBA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.031896234 (195h40m17.047565835s) [noise=true , address=45.91.116.9:4261 , piece_id=DQEJWL3R554ZEJILRN6CMJDUV7BWUNVG5NBYUSGKSAAZQOX2FZIQ, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.034277979 (195h40m17.049947586s) [noise=true , address=145.79.227.40:4054 , piece_id=JGZOLMWOGAMEXZUGGKPGO6OD5K3PBAAIZL2BRVVS4WDAJY2WUBJA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.035685433 (195h40m17.051355027s) [noise=true , address=145.79.227.47:4269 , piece_id=RGFY3HYS5X4LFIXS2C44F4M53WKBGJHMPYTGRHRVLC27WWKSXHKA, seg_pos={0 0}, upload=187956]: Uploading piece... Mar 18 19:51:14.631431078 (195h40m17.64710067s) [upload=187956]: Next returned nil segment Mar 18 19:51:14.631445772 (195h40m17.647115372s) [upload=187956]: Waiting for error group managing segments... Mar 18 19:51:14.774733936 (195h40m17.790403541s) [noise=true , address=45.91.116.12:4260 , piece_id=3TKSZPWPYFJFHUKS3UAY7BSVLNYHE5JLVIYF2QRFGAAWE52YUMPQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: <nil> Mar 18 19:51:14.779027565 (195h40m17.794697169s) [noise=true , address=45.91.116.13:4155 , piece_id=UG6JO6WWSQQ772OW4LCNH674XATGGP3AEDZHP6ZHS7SFVRI67CWA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: <nil> Mar 18 19:51:14.783357647 (195h40m17.799027249s) [noise=true , address=77.90.31.10:4519 , piece_id=FHF524X773L3ZTFOGROTJCQGXLBWGVKLSTHZDGLGBUJQAEHDWZ5A, seg_pos={0 0}, upload=187956]: Done uploading piece. err: <nil> Mar 18 19:51:14.806018966 (195h40m17.821688571s) [noise=true , address=79.127.219.45:4854 , piece_id=QVHKP5MKEWB3S5SKVI7425UPHMYV4UKYKO4GN25DDIJL4DIS4JKQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: <nil> Mar 18 19:51:14.813051122 (195h40m17.828720722s) [noise=true , address=45.91.116.9:4261 , piece_id=DQEJWL3R554ZEJILRN6CMJDUV7BWUNVG5NBYUSGKSAAZQOX2FZIQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: <nil> Mar 18 19:51:39.636097802 (195h40m42.651767413s) [noise=true , address=145.79.227.3:4153 , piece_id=EWO6MGOBTUNM7LRDEBQKMRNAO2EMOMTSJ7WKXMUKWDUDDN2EIZRA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:124myE1wFhvjA5paJy5tr2hRe8jZCoEdbJLdBwvss1ZnT4GvtZv, address:145.79.227.3:4153): protocol: rpc: dial tcp 145.79.227.3:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.636509160 (195h40m42.652178776s) [noise=true , address=145.79.227.2:4119 , piece_id=NREBXP2UBDE4DO44LFUTKOISJXM2QFAKX7M65JCJCVDRYOZTOPIQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:145.79.227.2:4119): protocol: rpc: dial tcp 145.79.227.2:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.636583494 (195h40m42.652253114s) [noise=true , address=145.79.227.47:4269 , piece_id=RGFY3HYS5X4LFIXS2C44F4M53WKBGJHMPYTGRHRVLC27WWKSXHKA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:12B63jT3ne2EnUiH7eRFWMpjzdirh6u7PwprmTUmDHPquoQwgk9, address:145.79.227.47:4269): protocol: rpc: dial tcp 145.79.227.47:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.636778990 (195h40m42.652448602s) [noise=true , address=145.79.227.44:4198 , piece_id=6DZALCMGRZTL5WLUSI6RFTPNS3MPI4PW35K4JOSZENYMECLM4QTA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:145.79.227.44:4198): protocol: rpc: dial tcp 145.79.227.44:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.636888179 (195h40m42.652557794s) [noise=true , address=145.79.227.51:4410 , piece_id=7IMSVXUH22BHYLVJV66J6PFKOKU23VMWQMLISIN67XUBMDZGLBMA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: rpc: dial tcp 145.79.227.51:4410: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.636954169 (195h40m42.652623782s) [noise=true , address=145.79.227.40:4054 , piece_id=JGZOLMWOGAMEXZUGGKPGO6OD5K3PBAAIZL2BRVVS4WDAJY2WUBJA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v, address:145.79.227.40:4054): protocol: rpc: dial tcp 145.79.227.40:4054: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637105972 (195h40m42.652775587s) [noise=true , address=145.79.227.5:4226 , piece_id=OYKQ23W5OPHZV3MHPDTUC2D4RUWOFY2AKO3YAGRYASUS2AYJOPJQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:145.79.227.5:4226): protocol: rpc: dial tcp 145.79.227.5:4226: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637209440 (195h40m42.652879052s) [noise=true , address=145.79.227.25:4619 , piece_id=N35DZJCJNH6EHHOBFA2UQUTYWJ6DCY54OIDVIEYQZ3AILNKUIJEQ, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: rpc: dial tcp 145.79.227.25:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637289088 (195h40m42.6529587s) [noise=true , address=145.79.227.12:4771 , piece_id=EPSFCRGJC374HJBYCXJEHW3JKD6BUNURQ466A5WOUQW2DRHA2MYA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:145.79.227.12:4771): protocol: rpc: dial tcp 145.79.227.12:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637349412 (195h40m42.653019026s) [noise=true , address=145.79.227.13:4806 , piece_id=MG47RNSKRGHSMRHA7ZD4XV3ODZD72DSJHYU27XCOYOBDKAQWSVHA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:12qUyweKQhw15RjewCgKi1oLyCvN3VUB1VMpeGhj2Q2hszvJBsQ, address:145.79.227.13:4806): protocol: rpc: dial tcp 145.79.227.13:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637419473 (195h40m42.653089089s) [noise=true , address=145.79.227.45:4233 , piece_id=KC7IHYX32KX3AECFS2HUKZMU766BWY3MPGLNW5H5IUEHJ7OJZDBA, seg_pos={0 0}, upload=187956]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:145.79.227.45:4233): protocol: rpc: dial tcp 145.79.227.45:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:51:39.637462412 (195h40m42.653132015s) [seg_pos={0 0}, upload=187956]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:51:39.750129298 (195h40m42.765798936s) [upload=187956]: Done uploading segments... Mar 18 19:51:39.750198713 (195h40m42.76586832s) [upload=187956]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.117026347s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....................................XXXXXXXX....XX.XXXXXXXXX] duration: 5.005576326s trace: https://jaeger.dev.storj.io/trace/13ba84d5c9f5bab2 FAIL (2026-03-18 19:51:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.005601451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXX.XXX.............] duration: 20.819987914s trace: https://jaeger.dev.storj.io/trace/4d6a8ab787f335c0 FAIL (2026-03-18 19:51:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.820013448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X..X...X..X..X........] duration: 20.431693243s trace: https://jaeger.dev.storj.io/trace/220541cf08cfd830 FAIL (2026-03-18 19:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .BatchRequestItem_ObjectBegin Mar 18 19:45:25.326452964 (195h34m28.342122568s) [upload=187860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:45:25.456683819 (195h34m28.472353428s) [seg_pos={0 0}, upload=187860]: Begin upload segment... Mar 18 19:45:25.456770594 (195h34m28.472440184s) [seg_pos={0 0}, upload=187860]: Done begin upload segment. Mar 18 19:45:25.456821582 (195h34m28.472491172s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.456867918 (195h34m28.472537526s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.457527075 (195h34m28.47319668s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458341322 (195h34m28.474010934s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458481997 (195h34m28.4741516s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459053712 (195h34m28.474723318s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459541420 (195h34m28.475211026s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460127026 (195h34m28.475796632s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460537480 (195h34m28.476207086s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461390949 (195h34m28.477060561s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461470344 (195h34m28.477139959s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462470296 (195h34m28.478139902s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462553305 (195h34m28.478222915s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.463505826 (195h34m28.479175427s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.464069344 (195h34m28.479738951s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465380624 (195h34m28.481050227s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465888656 (195h34m28.481558265s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.466308049 (195h34m28.481977655s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:26.019613415 (195h34m29.035283015s) [upload=187860]: Next returned nil segment Mar 18 19:45:26.019627850 (195h34m29.035297451s) [upload=187860]: Waiting for error group managing segments... Mar 18 19:45:26.193532210 (195h34m29.209201815s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.218509951 (195h34m29.234179558s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.219073762 (195h34m29.234743372s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.271321232 (195h34m29.286990847s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.293176147 (195h34m29.308845754s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.662025397 (195h34m29.677694999s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.879968454 (195h34m29.895638055s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.898670905 (195h34m29.914340506s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.903771534 (195h34m29.919441134s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.993657693 (195h34m30.009327292s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:46:25.039034405 (195h35m28.054704015s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039411284 (195h35m28.055080905s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:145.79.227.44:4206): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039509444 (195h35m28.055179057s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039966227 (195h35m28.055635833s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:145.79.227.33:4887): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040027384 (195h35m28.05569699s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1xnx9shwwZAcWVV6aPj2ikukj8nbKiFsBs1hiS89zSkRQqNMJD, address:145.79.227.17:4993): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040345443 (195h35m28.056015047s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12GACnovbrme3fQknSDpZWVLq838VKop9mPjxnVXGdJ8KaqyaTi, address:145.79.227.20:4472): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040426406 (195h35m28.056095999s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12mkHFDS6YCheghJdaJFybhcv9yhV4Acfs1RhnCW8yzA23neFRC, address:145.79.227.11:4747): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040650249 (195h35m28.056319864s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:145.79.227.36:4525): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040699435 (195h35m28.056369021s) [seg_pos={0 0}, upload=187860]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Mar 18 19:46:25.162188571 (195h35m28.177858163s) [upload=187860]: Done uploading segments... Mar 18 19:46:25.162240944 (195h35m28.177910547s) [upload=187860]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 20.431727119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.113927075s trace: https://jaeger.dev.storj.io/trace/420d920a5973be65 FAIL (2026-03-18 19:50:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 87945]: Begin upload segment... Mar 18 19:50:25.777463017 (195h39m28.793132626s) [noise=true , address=145.79.227.47:4266 , piece_id=4KE5LDCQCXVVITL57BWCYDQ6BNPEQJFDEZRAZCFORSHYKGAQM3BQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.777970018 (195h39m28.793639639s) [seg_pos={0 0}, upload=187945]: Done begin upload segment. Mar 18 19:50:25.778021245 (195h39m28.79369085s) [noise=true , address=145.79.227.19:4078 , piece_id=LQRY2VJY3SMILSIMDYOZJ7Z7BYJ3ZUTA3UDKBOWROBBYSK436X5A, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.778445308 (195h39m28.794114911s) [noise=true , address=145.79.227.20:4440 , piece_id=VLFBRRQDCA7MSFPWNTFXPK5TUBYULNFLVFLSGI3PLHZHMAQCMZXQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.778767952 (195h39m28.794437555s) [noise=true , address=45.91.116.5:4299 , piece_id=STEV56YKKD54UFSD76XV75EMNBHEZ6DK7NA7DAEBRKTG5F32BQKQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.779049273 (195h39m28.794718905s) [noise=true , address=145.79.227.46:4603 , piece_id=OONKW4XQVZLEOQVGQ7J75FPCFYWLRIY3GWNIXITWEOUKTTBG7XYA, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.779120141 (195h39m28.79478974s) [noise=true , address=145.79.227.17:4974 , piece_id=XNIJRD5TFTLIWIX5HCBU7MNOXMQN63RV5GSNGMUVR5VIBC2LE4YA, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.779481900 (195h39m28.795151504s) [noise=true , address=145.79.227.28:4731 , piece_id=UCM26X72LFTATDQ33UXZALNA7L7CK422WQWSOXH4RWQTLNNXOI4A, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.779722834 (195h39m28.795392444s) [noise=true , address=145.79.227.24:4585 , piece_id=O6CEL6UKVKX32RBL2JWWBCXTZPDJ4CSX6D4W3JA2RWHK3XXFID4A, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.779875808 (195h39m28.795545411s) [noise=true , address=145.79.227.18:4011 , piece_id=P4KYFHUKEU7X6ISBDVMND4LAMA7QFMND7RID43OLDTESVZFGBRDQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.780284442 (195h39m28.795954054s) [noise=true , address=145.79.227.50:4374 , piece_id=WLDXGWLY3AZT2IVDAOQGU3TRF2XDXXXDRGZVYIKS42KPVS3JGS5A, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.780331335 (195h39m28.796000946s) [noise=true , address=45.91.116.13:4188 , piece_id=N43EMSYIWRYBURCMN6SHXLG5WQHMS5THZRSVU4K4TQ5K3E4VVWVA, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.780660079 (195h39m28.796329682s) [noise=true , address=45.91.116.10:4081 , piece_id=TPRG74MHON5QEKQMZ4QMKGTKQRZUMVUGJGM7TCYZKAKSQRXCLX5Q, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.780881865 (195h39m28.796551474s) [noise=true , address=77.90.31.11:4553 , piece_id=BH64XLVW7ITICTF6WE4HGEI37VKO24OFH47XV3V4OCDA6DV5SC5A, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.781050354 (195h39m28.796719961s) [noise=true , address=145.79.227.45:4234 , piece_id=EINXCC3GZEHEO55HHJ2PY75RNGZ76JGAPCFQ5NDJY3C4ADR4JPNQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.781452058 (195h39m28.797121662s) [noise=true , address=77.90.31.21:4880 , piece_id=2QKOLN4DMBWFCNB4UZGPYYPLJ24IQ5SJ43QQIEVBNCZYLSMZC77Q, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:25.781564243 (195h39m28.797233855s) [noise=true , address=145.79.227.11:4734 , piece_id=QRQU26TSHGTHXIIRBD7XULPCMIQSDSNFEQ4XBSONAU2ZODJW62KQ, seg_pos={0 0}, upload=187945]: Uploading piece... Mar 18 19:50:26.380725420 (195h39m29.396395007s) [upload=187945]: Next returned nil segment Mar 18 19:50:26.380738904 (195h39m29.396408504s) [upload=187945]: Waiting for error group managing segments... Mar 18 19:50:26.521147415 (195h39m29.536817021s) [noise=true , address=77.90.31.21:4880 , piece_id=2QKOLN4DMBWFCNB4UZGPYYPLJ24IQ5SJ43QQIEVBNCZYLSMZC77Q, seg_pos={0 0}, upload=187945]: Done uploading piece. err: <nil> Mar 18 19:50:26.524652903 (195h39m29.540322511s) [noise=true , address=45.91.116.13:4188 , piece_id=N43EMSYIWRYBURCMN6SHXLG5WQHMS5THZRSVU4K4TQ5K3E4VVWVA, seg_pos={0 0}, upload=187945]: Done uploading piece. err: <nil> Mar 18 19:50:26.529226165 (195h39m29.544895778s) [noise=true , address=77.90.31.11:4553 , piece_id=BH64XLVW7ITICTF6WE4HGEI37VKO24OFH47XV3V4OCDA6DV5SC5A, seg_pos={0 0}, upload=187945]: Done uploading piece. err: <nil> Mar 18 19:50:26.531678687 (195h39m29.547348297s) [noise=true , address=45.91.116.5:4299 , piece_id=STEV56YKKD54UFSD76XV75EMNBHEZ6DK7NA7DAEBRKTG5F32BQKQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: <nil> Mar 18 19:50:26.534079157 (195h39m29.549748759s) [noise=true , address=45.91.116.10:4081 , piece_id=TPRG74MHON5QEKQMZ4QMKGTKQRZUMVUGJGM7TCYZKAKSQRXCLX5Q, seg_pos={0 0}, upload=187945]: Done uploading piece. err: <nil> Mar 18 19:50:51.332056344 (195h39m54.347725948s) [noise=true , address=145.79.227.45:4234 , piece_id=EINXCC3GZEHEO55HHJ2PY75RNGZ76JGAPCFQ5NDJY3C4ADR4JPNQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: rpc: dial tcp 145.79.227.45:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.332413611 (195h39m54.348083214s) [noise=true , address=145.79.227.11:4734 , piece_id=QRQU26TSHGTHXIIRBD7XULPCMIQSDSNFEQ4XBSONAU2ZODJW62KQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: rpc: dial tcp 145.79.227.11:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.332726185 (195h39m54.348395796s) [noise=true , address=145.79.227.47:4266 , piece_id=4KE5LDCQCXVVITL57BWCYDQ6BNPEQJFDEZRAZCFORSHYKGAQM3BQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, address:145.79.227.47:4266): protocol: rpc: dial tcp 145.79.227.47:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.333145376 (195h39m54.348814987s) [noise=true , address=145.79.227.19:4078 , piece_id=LQRY2VJY3SMILSIMDYOZJ7Z7BYJ3ZUTA3UDKBOWROBBYSK436X5A, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: rpc: dial tcp 145.79.227.19:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.333466609 (195h39m54.349136219s) [noise=true , address=145.79.227.20:4440 , piece_id=VLFBRRQDCA7MSFPWNTFXPK5TUBYULNFLVFLSGI3PLHZHMAQCMZXQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.334029816 (195h39m54.349699427s) [noise=true , address=145.79.227.46:4603 , piece_id=OONKW4XQVZLEOQVGQ7J75FPCFYWLRIY3GWNIXITWEOUKTTBG7XYA, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.334449587 (195h39m54.350119189s) [noise=true , address=145.79.227.28:4731 , piece_id=UCM26X72LFTATDQ33UXZALNA7L7CK422WQWSOXH4RWQTLNNXOI4A, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK, address:145.79.227.28:4731): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:50:51.334508592 (195h39m54.350178272s) [noise=true , address=145.79.227.24:4585 , piece_id=O6CEL6UKVKX32RBL2JWWBCXTZPDJ4CSX6D4W3JA2RWHK3XXFID4A, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.334764713 (195h39m54.350434323s) [noise=true , address=145.79.227.18:4011 , piece_id=P4KYFHUKEU7X6ISBDVMND4LAMA7QFMND7RID43OLDTESVZFGBRDQ, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:145.79.227.18:4011): protocol: rpc: dial tcp 145.79.227.18:4011: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.334853007 (195h39m54.350522631s) [noise=true , address=145.79.227.17:4974 , piece_id=XNIJRD5TFTLIWIX5HCBU7MNOXMQN63RV5GSNGMUVR5VIBC2LE4YA, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1, address:145.79.227.17:4974): protocol: rpc: dial tcp 145.79.227.17:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.335070613 (195h39m54.350740225s) [noise=true , address=145.79.227.50:4374 , piece_id=WLDXGWLY3AZT2IVDAOQGU3TRF2XDXXXDRGZVYIKS42KPVS3JGS5A, seg_pos={0 0}, upload=187945]: Done uploading piece. err: ecclient: upload failed (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K, address:145.79.227.50:4374): protocol: rpc: dial tcp 145.79.227.50:4374: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:51.335123711 (195h39m54.350793313s) [seg_pos={0 0}, upload=187945]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Mar 18 19:50:51.443933281 (195h39m54.459602888s) [upload=187945]: Done uploading segments... Mar 18 19:50:51.443994437 (195h39m54.459664048s) [upload=187945]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.113944764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X..X...X..X..X.......] duration: 23.884003871s trace: https://jaeger.dev.storj.io/trace/220541cf08cfd830 FAIL (2026-03-18 19:50:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .BatchRequestItem_ObjectBegin Mar 18 19:45:25.326452964 (195h34m28.342122568s) [upload=187860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:45:25.456683819 (195h34m28.472353428s) [seg_pos={0 0}, upload=187860]: Begin upload segment... Mar 18 19:45:25.456770594 (195h34m28.472440184s) [seg_pos={0 0}, upload=187860]: Done begin upload segment. Mar 18 19:45:25.456821582 (195h34m28.472491172s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.456867918 (195h34m28.472537526s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.457527075 (195h34m28.47319668s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458341322 (195h34m28.474010934s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458481997 (195h34m28.4741516s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459053712 (195h34m28.474723318s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459541420 (195h34m28.475211026s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460127026 (195h34m28.475796632s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460537480 (195h34m28.476207086s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461390949 (195h34m28.477060561s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461470344 (195h34m28.477139959s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462470296 (195h34m28.478139902s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462553305 (195h34m28.478222915s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.463505826 (195h34m28.479175427s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.464069344 (195h34m28.479738951s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465380624 (195h34m28.481050227s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465888656 (195h34m28.481558265s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.466308049 (195h34m28.481977655s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:26.019613415 (195h34m29.035283015s) [upload=187860]: Next returned nil segment Mar 18 19:45:26.019627850 (195h34m29.035297451s) [upload=187860]: Waiting for error group managing segments... Mar 18 19:45:26.193532210 (195h34m29.209201815s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.218509951 (195h34m29.234179558s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.219073762 (195h34m29.234743372s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.271321232 (195h34m29.286990847s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.293176147 (195h34m29.308845754s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.662025397 (195h34m29.677694999s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.879968454 (195h34m29.895638055s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.898670905 (195h34m29.914340506s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.903771534 (195h34m29.919441134s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.993657693 (195h34m30.009327292s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:46:25.039034405 (195h35m28.054704015s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039411284 (195h35m28.055080905s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:145.79.227.44:4206): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039509444 (195h35m28.055179057s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039966227 (195h35m28.055635833s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:145.79.227.33:4887): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040027384 (195h35m28.05569699s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1xnx9shwwZAcWVV6aPj2ikukj8nbKiFsBs1hiS89zSkRQqNMJD, address:145.79.227.17:4993): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040345443 (195h35m28.056015047s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12GACnovbrme3fQknSDpZWVLq838VKop9mPjxnVXGdJ8KaqyaTi, address:145.79.227.20:4472): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040426406 (195h35m28.056095999s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12mkHFDS6YCheghJdaJFybhcv9yhV4Acfs1RhnCW8yzA23neFRC, address:145.79.227.11:4747): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040650249 (195h35m28.056319864s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:145.79.227.36:4525): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040699435 (195h35m28.056369021s) [seg_pos={0 0}, upload=187860]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Mar 18 19:46:25.162188571 (195h35m28.177858163s) [upload=187860]: Done uploading segments... Mar 18 19:46:25.162240944 (195h35m28.177910547s) [upload=187860]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 23.884024105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.....................................XXXXXXXX....XX.XXXXXXXX] duration: 5.000748789s trace: https://jaeger.dev.storj.io/trace/491460bbabd4cdcc FAIL (2026-03-18 19:50:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00077767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.10964283s trace: https://jaeger.dev.storj.io/trace/2340eba8b79325d FAIL (2026-03-18 19:50:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Mar 18 19:49:51.354023327 (195h38m54.369692933s) [upload=187938]: Starting upload Mar 18 19:49:51.354280847 (195h38m54.369950448s) [upload=187938]: Done starting upload Mar 18 19:49:51.354405161 (195h38m54.370074763s) [upload=187938]: Uploading segments... Mar 18 19:49:51.354984071 (195h38m54.370653687s) [upload=187938]: Got next segment. Inline: false Mar 18 19:49:51.355102470 (195h38m54.37077212s) [upload=187938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:49:51.355112330 (195h38m54.370781943s) [upload=187938]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:49:55.527161184 (195h38m58.542830784s) [seg_pos={0 0}, upload=187938]: Begin upload segment... Mar 18 19:49:55.527212041 (195h38m58.542881641s) [seg_pos={0 0}, upload=187938]: Done begin upload segment. Mar 18 19:49:55.527240935 (195h38m58.542910536s) [noise=true , address=145.79.227.49:4342 , piece_id=KQ4XMOLZTQJVMZ43RLJR2SVB6F4B5KBHKAFTX6HI5FLBCQCSXXLA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.527299890 (195h38m58.542969491s) [noise=true , address=79.127.219.38:4822 , piece_id=LNAUBQLGR66FDIKQUHHPPCYSUPRH24PACM2RFH5CX4BRSW6OALEA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.528132958 (195h38m58.543802565s) [noise=true , address=77.90.31.15:4700 , piece_id=6AYXZHXAW33ZSPNWOUEPARXZLJ53J55IAM4BIIJG2NXFDY6FMLXA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.528233485 (195h38m58.543903088s) [noise=true , address=145.79.227.35:4486 , piece_id=AXHEGSG7RFGNTBZVC6MI6MIRSW4SWWSB423CINCE4ZDADNC7AQ3Q, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.528703920 (195h38m58.54437352s) [noise=true , address=77.90.31.14:4666 , piece_id=A6MHXTZO4FCNUPYZQBTFCCMIK5SMQGS7PGFR7LXA5N2T66AP6JGA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.529046836 (195h38m58.544716487s) [noise=true , address=145.79.227.21:4473 , piece_id=J574VT6LEAZJH7MFCMR55RZFLVOJUX4S62SGX5MUSNWT2QIC65BQ, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.529389842 (195h38m58.545059445s) [noise=true , address=77.90.31.10:4522 , piece_id=M4R7EJYD76BS7QGLLV3NMRC7C6URTIJCLVYMJCNUSFZCSNCCQ5HQ, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.529952400 (195h38m58.545622001s) [noise=true , address=145.79.227.33:4873 , piece_id=DKZCKPTUPOIHLR6OETYCH2RIOVVIR3MVMK4PS37RRESIXL2S5MSA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.530402569 (195h38m58.546072169s) [noise=true , address=77.90.31.21:4877 , piece_id=UMHYRQVDZGVXX54E6SEZWRMW4XYNIT5GAAE4CKBHWVZ47QYFGSOA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.530428937 (195h38m58.546098539s) [noise=true , address=79.127.219.45:4855 , piece_id=46FEVI3TSSYEAWTUENBXGWHXTDRU3G7VH56TDQZU7JC2BA57AT4A, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.530880085 (195h38m58.546549693s) [noise=true , address=77.90.31.19:4844 , piece_id=QHW2WVWOX6SUPWAIAL4QAGACN5WYUX636XM6VXPRDRXWQZRIL22A, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.530888270 (195h38m58.546557876s) [noise=true , address=45.91.116.8:4049 , piece_id=JKCHWQRQX5AOZ3MMHRYUUC3GD7345XXBG2ZBOPKAL6TOQQTTKMYA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.531286844 (195h38m58.546956447s) [noise=true , address=145.79.227.6:4261 , piece_id=DY5WETC4WV6Q3DSEROLUYNOKT6O3V25GHHEI2SEMJQVUTHX4ARHA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.531320836 (195h38m58.546990445s) [noise=true , address=145.79.227.50:4376 , piece_id=BPGGRVTPTI5NRZJL24JXNBTCCWRFP7P3HI4RPI4EF64VZUV5YEWA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.531729438 (195h38m58.547399056s) [noise=true , address=145.79.227.29:4945 , piece_id=DVXVKPW7UTZ22BNBVYP6BYBCX6O56QKADIMAJIGDKILR35O6S7VQ, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:55.532182756 (195h38m58.547852362s) [noise=true , address=77.90.31.17:4770 , piece_id=67SUTAVY2CZVMPXOKNEIOA27PVNZZYBCD55CIJMGCS6M5TWYWEBA, seg_pos={0 0}, upload=187938]: Uploading piece... Mar 18 19:49:56.126548747 (195h38m59.142218347s) [upload=187938]: Next returned nil segment Mar 18 19:49:56.126562003 (195h38m59.142231608s) [upload=187938]: Waiting for error group managing segments... Mar 18 19:49:56.254497763 (195h38m59.27016737s) [noise=true , address=77.90.31.19:4844 , piece_id=QHW2WVWOX6SUPWAIAL4QAGACN5WYUX636XM6VXPRDRXWQZRIL22A, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.255022331 (195h38m59.270691955s) [noise=true , address=77.90.31.21:4877 , piece_id=UMHYRQVDZGVXX54E6SEZWRMW4XYNIT5GAAE4CKBHWVZ47QYFGSOA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.272398023 (195h38m59.288067623s) [noise=true , address=77.90.31.17:4770 , piece_id=67SUTAVY2CZVMPXOKNEIOA27PVNZZYBCD55CIJMGCS6M5TWYWEBA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.289400346 (195h38m59.305069971s) [noise=true , address=77.90.31.15:4700 , piece_id=6AYXZHXAW33ZSPNWOUEPARXZLJ53J55IAM4BIIJG2NXFDY6FMLXA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.289553979 (195h38m59.305223578s) [noise=true , address=77.90.31.10:4522 , piece_id=M4R7EJYD76BS7QGLLV3NMRC7C6URTIJCLVYMJCNUSFZCSNCCQ5HQ, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.289868888 (195h38m59.305538489s) [noise=true , address=77.90.31.14:4666 , piece_id=A6MHXTZO4FCNUPYZQBTFCCMIK5SMQGS7PGFR7LXA5N2T66AP6JGA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.290270379 (195h38m59.30593998s) [noise=true , address=45.91.116.8:4049 , piece_id=JKCHWQRQX5AOZ3MMHRYUUC3GD7345XXBG2ZBOPKAL6TOQQTTKMYA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.323685709 (195h38m59.339355314s) [noise=true , address=79.127.219.45:4855 , piece_id=46FEVI3TSSYEAWTUENBXGWHXTDRU3G7VH56TDQZU7JC2BA57AT4A, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:49:56.378011584 (195h38m59.393681185s) [noise=true , address=79.127.219.38:4822 , piece_id=LNAUBQLGR66FDIKQUHHPPCYSUPRH24PACM2RFH5CX4BRSW6OALEA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: <nil> Mar 18 19:50:21.072070701 (195h39m24.087740301s) [noise=true , address=145.79.227.21:4473 , piece_id=J574VT6LEAZJH7MFCMR55RZFLVOJUX4S62SGX5MUSNWT2QIC65BQ, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: rpc: dial tcp 145.79.227.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.072506802 (195h39m24.088176413s) [noise=true , address=145.79.227.33:4873 , piece_id=DKZCKPTUPOIHLR6OETYCH2RIOVVIR3MVMK4PS37RRESIXL2S5MSA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:145.79.227.33:4873): protocol: rpc: dial tcp 145.79.227.33:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.072977452 (195h39m24.088647055s) [noise=true , address=145.79.227.6:4261 , piece_id=DY5WETC4WV6Q3DSEROLUYNOKT6O3V25GHHEI2SEMJQVUTHX4ARHA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:1i1jcJdEP7Ch9uaGcjDaVxpodx55dJEHUKFbcr1GeKQ9waWjYz, address:145.79.227.6:4261): protocol: rpc: dial tcp 145.79.227.6:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.073291372 (195h39m24.088960984s) [noise=true , address=145.79.227.35:4486 , piece_id=AXHEGSG7RFGNTBZVC6MI6MIRSW4SWWSB423CINCE4ZDADNC7AQ3Q, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:145.79.227.35:4486): protocol: rpc: dial tcp 145.79.227.35:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.073818553 (195h39m24.089488163s) [noise=true , address=145.79.227.49:4342 , piece_id=KQ4XMOLZTQJVMZ43RLJR2SVB6F4B5KBHKAFTX6HI5FLBCQCSXXLA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:145.79.227.49:4342): protocol: rpc: dial tcp 145.79.227.49:4342: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.074213079 (195h39m24.089882689s) [noise=true , address=145.79.227.50:4376 , piece_id=BPGGRVTPTI5NRZJL24JXNBTCCWRFP7P3HI4RPI4EF64VZUV5YEWA, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:12MnvtjSKHuNKUcVEjTviEGThRKQxzmGiMu1ofCwkv8BLAhGuzL, address:145.79.227.50:4376): protocol: rpc: dial tcp 145.79.227.50:4376: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.074336682 (195h39m24.090006288s) [noise=true , address=145.79.227.29:4945 , piece_id=DVXVKPW7UTZ22BNBVYP6BYBCX6O56QKADIMAJIGDKILR35O6S7VQ, seg_pos={0 0}, upload=187938]: Done uploading piece. err: ecclient: upload failed (node:128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, address:145.79.227.29:4945): protocol: rpc: dial tcp 145.79.227.29:4945: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:50:21.074375240 (195h39m24.090044831s) [seg_pos={0 0}, upload=187938]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Mar 18 19:50:21.178980755 (195h39m24.194650362s) [upload=187938]: Done uploading segments... Mar 18 19:50:21.179025935 (195h39m24.194695543s) [upload=187938]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.109666514s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXX.XXX............] duration: 18.473228258s trace: https://jaeger.dev.storj.io/trace/4d6a8ab787f335c0 FAIL (2026-03-18 19:50:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.473260068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................X..X...X..X..X......] duration: 36.092903055s trace: https://jaeger.dev.storj.io/trace/220541cf08cfd830 FAIL (2026-03-18 19:50:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== .BatchRequestItem_ObjectBegin Mar 18 19:45:25.326452964 (195h34m28.342122568s) [upload=187860]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:45:25.456683819 (195h34m28.472353428s) [seg_pos={0 0}, upload=187860]: Begin upload segment... Mar 18 19:45:25.456770594 (195h34m28.472440184s) [seg_pos={0 0}, upload=187860]: Done begin upload segment. Mar 18 19:45:25.456821582 (195h34m28.472491172s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.456867918 (195h34m28.472537526s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.457527075 (195h34m28.47319668s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458341322 (195h34m28.474010934s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.458481997 (195h34m28.4741516s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459053712 (195h34m28.474723318s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.459541420 (195h34m28.475211026s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460127026 (195h34m28.475796632s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.460537480 (195h34m28.476207086s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461390949 (195h34m28.477060561s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.461470344 (195h34m28.477139959s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462470296 (195h34m28.478139902s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.462553305 (195h34m28.478222915s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.463505826 (195h34m28.479175427s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.464069344 (195h34m28.479738951s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465380624 (195h34m28.481050227s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.465888656 (195h34m28.481558265s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:25.466308049 (195h34m28.481977655s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Uploading piece... Mar 18 19:45:26.019613415 (195h34m29.035283015s) [upload=187860]: Next returned nil segment Mar 18 19:45:26.019627850 (195h34m29.035297451s) [upload=187860]: Waiting for error group managing segments... Mar 18 19:45:26.193532210 (195h34m29.209201815s) [noise=true , address=45.91.116.5:4324 , piece_id=ZFBWMYRD65NBX2FUNBIJF57VRTZOUJEPXOEWHWU63QWBATPWJCDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.218509951 (195h34m29.234179558s) [noise=true , address=79.127.219.38:4827 , piece_id=2WC4XR64CXG7I7EPU6VFNYUAC76BJ3EKEPKSPRT4I55IYRFQVXLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.219073762 (195h34m29.234743372s) [noise=true , address=77.90.31.10:4537 , piece_id=N4UDTJUBLM6NLN2NNE3YYKAYHMCNGQWLX4R6TZ35UPDONZ7NMIOA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.271321232 (195h34m29.286990847s) [noise=true , address=79.127.219.46:4920 , piece_id=76VAJBNCJC4WAYXSEEQGZW7BEGGULKJFLV75UOBDBAVII7AKR62A, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.293176147 (195h34m29.308845754s) [noise=true , address=77.90.31.17:4782 , piece_id=7ZJZOZOLIXGWYAGAV65WIEY76DOCM6CSE2HAUWXRPKHCLVOQQFEQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.662025397 (195h34m29.677694999s) [noise=true , address=167.17.72.69:4716 , piece_id=3ETXEX7UWPM4DU3DL2EGA5Z4QLKUKHAT23KG3PD5YL6XEZCJTGRA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.879968454 (195h34m29.895638055s) [noise=true , address=79.127.205.233:4144 , piece_id=C3QUG3TRTWKVN3ZOCEWVNP74WE4QMYA65AHCAKEXOPBPKY4AUZDA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.898670905 (195h34m29.914340506s) [noise=true , address=79.127.205.235:4260 , piece_id=VRXKZO7BI77HQO7F33I3EHJZQOS3VCNGH4XZGZEEOHYTM4FOIIDQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.903771534 (195h34m29.919441134s) [noise=true , address=193.149.164.68:4039 , piece_id=FLJZMLJXBWFG7FOEHWTVDJU2NTW2GZPHNK5S2MIT5K2EBEMACKSQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:45:26.993657693 (195h34m30.009327292s) [noise=true , address=79.127.205.234:4206 , piece_id=FCSOPKSPRRYZ5E6RDK72WZVVAWTOD53MYSUST63NR52RY6QDHMKQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: <nil> Mar 18 19:46:25.039034405 (195h35m28.054704015s) [noise=true , address=145.79.227.2:4135 , piece_id=DHYVS2YDQW7FQANTEAPJKWBCNLUWPZNANCHFNDWLGFHPVOYPVTJQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:145.79.227.2:4135): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039411284 (195h35m28.055080905s) [noise=true , address=145.79.227.44:4206 , piece_id=L42NBORHYROKARBF6Z7CUJQ4SO5IA5SOBRA3TJWOYZDZKTSNXE4Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12Mp4ZzYgrP3ZSXDeHbSTXaAJMPgGKUqVjdNtyfeKE5qBMAzBke, address:145.79.227.44:4206): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039509444 (195h35m28.055179057s) [noise=true , address=145.79.227.5:4238 , piece_id=PNXIWK2VZZSCR53VQOBOMP5Z4F5YYRIWWX4CNPOXNRJAB4RWKJLQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:122AVjaN3YaiABnUjuZwSb6apPDeszZAWvmzwtP47k1fxsKLD5V, address:145.79.227.5:4238): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.039966227 (195h35m28.055635833s) [noise=true , address=145.79.227.33:4887 , piece_id=GPFUU4XAWYSCN4AG3VV6AI6BWEGLSHJFN5VGTVFGB3PFHPXOGU5Q, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:145.79.227.33:4887): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040027384 (195h35m28.05569699s) [noise=true , address=145.79.227.17:4993 , piece_id=AN3MJQVU5MVMQLL623PWSLWGH3N242LVPKFOG6AZAUJA6WGMHMFA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1xnx9shwwZAcWVV6aPj2ikukj8nbKiFsBs1hiS89zSkRQqNMJD, address:145.79.227.17:4993): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040345443 (195h35m28.056015047s) [noise=true , address=145.79.227.20:4472 , piece_id=6PS3R5ANRNMQFWPH3F2MWMIVTP53URHAQ7E3NLX72P6U6PGCLQTQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12GACnovbrme3fQknSDpZWVLq838VKop9mPjxnVXGdJ8KaqyaTi, address:145.79.227.20:4472): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040426406 (195h35m28.056095999s) [noise=true , address=145.79.227.11:4747 , piece_id=JTWQJUGUKHAVWGG6T2ZVJHZ4R5FZTTYFYKXXRD3MH4ZBLEOD4MRQ, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:12mkHFDS6YCheghJdaJFybhcv9yhV4Acfs1RhnCW8yzA23neFRC, address:145.79.227.11:4747): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040650249 (195h35m28.056319864s) [noise=true , address=145.79.227.36:4525 , piece_id=7FBEMPOKGXT2YB5Q6EEYJBFBX35RRCOOTQ2UETDSIVFW34CQZDVA, seg_pos={0 0}, upload=187860]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:145.79.227.36:4525): protocol: context deadline exceeded; context deadline exceeded Mar 18 19:46:25.040699435 (195h35m28.056369021s) [seg_pos={0 0}, upload=187860]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Mar 18 19:46:25.162188571 (195h35m28.177858163s) [upload=187860]: Done uploading segments... Mar 18 19:46:25.162240944 (195h35m28.177910547s) [upload=187860]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration:* 36.092930438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [......................................XXXXXXXX....XX.XXXXXXX] duration: 5.000842683s trace: https://jaeger.dev.storj.io/trace/7db1216f379bae9a FAIL (2026-03-18 19:50:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000858657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.......................................XXXXXXXX....XX.XXXXXX] duration: 5.000825558s trace: https://jaeger.dev.storj.io/trace/4eea55380056da36 FAIL (2026-03-18 19:49:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000925234s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXX.XXX...........] duration: 23.069262044s trace: https://jaeger.dev.storj.io/trace/4d6a8ab787f335c0 FAIL (2026-03-18 19:49:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.069309874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.117721351s trace: https://jaeger.dev.storj.io/trace/37d3400dfe202d90 FAIL (2026-03-18 19:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 49.210550461 (195h37m52.226220073s) [upload=187919]: Got next segment. Inline: false Mar 18 19:48:49.210610841 (195h37m52.226280455s) [upload=187919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 19:48:49.210616713 (195h37m52.226286328s) [upload=187919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 19:48:53.382213205 (195h37m56.397882817s) [seg_pos={0 0}, upload=187919]: Begin upload segment... Mar 18 19:48:53.382283831 (195h37m56.397953436s) [seg_pos={0 0}, upload=187919]: Done begin upload segment. Mar 18 19:48:53.382326872 (195h37m56.397996482s) [noise=true , address=77.90.31.17:4769 , piece_id=LHKGKJ67EY7QBR2U3WSMPSIEWNEGH37LZWJHBK6LS5ZYLGNJZGMQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.382412631 (195h37m56.398082237s) [noise=true , address=45.91.116.9:4264 , piece_id=ITEAOCO63WF3XS7FE7QHSM3H2J7BQNTZNMMV5IAXHDML3JI4TZWQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.383055977 (195h37m56.398725584s) [noise=true , address=145.79.227.38:4979 , piece_id=2SBCU2GXBFQLKQ3XFU2IMNEAPMHCZGKCNJZENY4HFE3OCQBKPR2A, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.383518331 (195h37m56.399187942s) [noise=true , address=79.127.219.45:4855 , piece_id=IZCFZZDVFZNJAGSGNHECWRVFF7UE4WO5RVLCG7RQRXCFK7IOPTFQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.383917811 (195h37m56.399587411s) [noise=true , address=145.79.227.37:4566 , piece_id=FBJZ7ZAPMUTTHNR3AL4IL7A5M3GNG5PPEOYMQQX44CN4SZZFY2LQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.384308262 (195h37m56.399977868s) [noise=true , address=145.79.227.10:4404 , piece_id=3W2G6JYQT33WVJPFVYDWCWVEO6SIKJP6NSNM5JA4WL3IN3QCSM3A, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.384697633 (195h37m56.400367232s) [noise=true , address=77.90.31.12:4593 , piece_id=CLXEKEVOGME5RJWHJHUFUCNSE2ZCBT7I4B3AOIKKD2TKA7OICMWQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.385044645 (195h37m56.400714249s) [noise=true , address=45.91.116.14:4216 , piece_id=AVCJKNVMWROBQIHKWUDQZVDF3IQLA6IGSSKYQU2MDMPKWLBTLQIQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.385382486 (195h37m56.40105209s) [noise=true , address=145.79.227.49:4339 , piece_id=DREBK5VXI4EWIUT73GW42BYU3ECO2EFOSQGRP4JQBMN3LT6PB2PA, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.386667093 (195h37m56.402336698s) [noise=true , address=145.79.227.48:4302 , piece_id=U3TDFGB2SCYHLIRDLO62K3X2DDRIOSKA7TQUUD2JRADRQUJZQFEQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.387172388 (195h37m56.402841999s) [noise=true , address=145.79.227.15:4875 , piece_id=DGBWAFY5F2K66PXJZ5E6NMYCBGEAHWOQYQYQGLTQBE6AIP34OMSQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.387621009 (195h37m56.403290611s) [noise=true , address=145.79.227.24:4586 , piece_id=44TF2YJO2W5YM655GIBUUAMD4JIUCYY64RK6M4JLKGQTTOV6JEJA, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.389365250 (195h37m56.405034856s) [noise=true , address=145.79.227.16:4915 , piece_id=RZADGX5LFIOELZYMMIA6S75AO5Q53QG53IF47RTCWI24D7XVX7SA, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.390222887 (195h37m56.405892488s) [noise=true , address=77.90.31.10:4518 , piece_id=XYD5TBU5UZ5WSCW6IWFRJGFMM3LW2TWAFOANUEAGRNJYCMDUZKQA, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.391485945 (195h37m56.407155556s) [noise=true , address=45.91.116.4:4955 , piece_id=ZEUFCFJE4VKGU4CDXAYP77ZGUMMP4RIQH667WH5IUEAQQCADT3WQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.393227551 (195h37m56.408897161s) [noise=true , address=145.79.227.34:4907 , piece_id=G2W3DLOTDSZQOZ4MGHRI54K7K3HM44VV6WU7NAZXDGF43ADSDBRQ, seg_pos={0 0}, upload=187919]: Uploading piece... Mar 18 19:48:53.994588325 (195h37m57.010257915s) [upload=187919]: Next returned nil segment Mar 18 19:48:53.994601803 (195h37m57.010271406s) [upload=187919]: Waiting for error group managing segments... Mar 18 19:48:54.136273639 (195h37m57.151943244s) [noise=true , address=45.91.116.9:4264 , piece_id=ITEAOCO63WF3XS7FE7QHSM3H2J7BQNTZNMMV5IAXHDML3JI4TZWQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.137336720 (195h37m57.153006321s) [noise=true , address=77.90.31.12:4593 , piece_id=CLXEKEVOGME5RJWHJHUFUCNSE2ZCBT7I4B3AOIKKD2TKA7OICMWQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.140316869 (195h37m57.155986469s) [noise=true , address=45.91.116.4:4955 , piece_id=ZEUFCFJE4VKGU4CDXAYP77ZGUMMP4RIQH667WH5IUEAQQCADT3WQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.141054317 (195h37m57.156723929s) [noise=true , address=45.91.116.14:4216 , piece_id=AVCJKNVMWROBQIHKWUDQZVDF3IQLA6IGSSKYQU2MDMPKWLBTLQIQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.146074000 (195h37m57.161743605s) [noise=true , address=77.90.31.17:4769 , piece_id=LHKGKJ67EY7QBR2U3WSMPSIEWNEGH37LZWJHBK6LS5ZYLGNJZGMQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.159412535 (195h37m57.175082143s) [noise=true , address=79.127.219.45:4855 , piece_id=IZCFZZDVFZNJAGSGNHECWRVFF7UE4WO5RVLCG7RQRXCFK7IOPTFQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:48:54.179253384 (195h37m57.194922985s) [noise=true , address=77.90.31.10:4518 , piece_id=XYD5TBU5UZ5WSCW6IWFRJGFMM3LW2TWAFOANUEAGRNJYCMDUZKQA, seg_pos={0 0}, upload=187919]: Done uploading piece. err: <nil> Mar 18 19:49:18.916295529 (195h38m21.931965134s) [noise=true , address=145.79.227.38:4979 , piece_id=2SBCU2GXBFQLKQ3XFU2IMNEAPMHCZGKCNJZENY4HFE3OCQBKPR2A, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:145.79.227.38:4979): protocol: rpc: dial tcp 145.79.227.38:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.916526185 (195h38m21.932195793s) [noise=true , address=145.79.227.48:4302 , piece_id=U3TDFGB2SCYHLIRDLO62K3X2DDRIOSKA7TQUUD2JRADRQUJZQFEQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, address:145.79.227.48:4302): protocol: rpc: dial tcp 145.79.227.48:4302: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.916706745 (195h38m21.932376351s) [noise=true , address=145.79.227.24:4586 , piece_id=44TF2YJO2W5YM655GIBUUAMD4JIUCYY64RK6M4JLKGQTTOV6JEJA, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:1mkxgR2oLXcTdyCsxmR9Q1an9ZeLgkTSHRCtR4ZJqKbsStNEMh, address:145.79.227.24:4586): protocol: rpc: dial tcp 145.79.227.24:4586: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.916751755 (195h38m21.932421368s) [noise=true , address=145.79.227.37:4566 , piece_id=FBJZ7ZAPMUTTHNR3AL4IL7A5M3GNG5PPEOYMQQX44CN4SZZFY2LQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, address:145.79.227.37:4566): protocol: rpc: dial tcp 145.79.227.37:4566: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.916967842 (195h38m21.93263745s) [noise=true , address=145.79.227.34:4907 , piece_id=G2W3DLOTDSZQOZ4MGHRI54K7K3HM44VV6WU7NAZXDGF43ADSDBRQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: rpc: dial tcp 145.79.227.34:4907: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.917142819 (195h38m21.932812431s) [noise=true , address=145.79.227.16:4915 , piece_id=RZADGX5LFIOELZYMMIA6S75AO5Q53QG53IF47RTCWI24D7XVX7SA, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: rpc: dial tcp 145.79.227.16:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.917283881 (195h38m21.932953493s) [noise=true , address=145.79.227.15:4875 , piece_id=DGBWAFY5F2K66PXJZ5E6NMYCBGEAHWOQYQYQGLTQBE6AIP34OMSQ, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.917487960 (195h38m21.933157561s) [noise=true , address=145.79.227.10:4404 , piece_id=3W2G6JYQT33WVJPFVYDWCWVEO6SIKJP6NSNM5JA4WL3IN3QCSM3A, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:12u1v9cAH2841Zq5JieXcPKWHYuEkg18S89ykFQuhtYBmvAPS7b, address:145.79.227.10:4404): protocol: rpc: dial tcp 145.79.227.10:4404: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.917999906 (195h38m21.933669509s) [noise=true , address=145.79.227.49:4339 , piece_id=DREBK5VXI4EWIUT73GW42BYU3ECO2EFOSQGRP4JQBMN3LT6PB2PA, seg_pos={0 0}, upload=187919]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:145.79.227.49:4339): protocol: rpc: dial tcp 145.79.227.49:4339: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 19:49:18.918060182 (195h38m21.933729775s) [seg_pos={0 0}, upload=187919]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Mar 18 19:49:19.031475406 (195h38m22.047145007s) [upload=187919]: Done uploading segments... Mar 18 19:49:19.031513424 (195h38m22.047183028s) [upload=187919]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.117793404s core uplink us-select-1

1
...
574
575
576
577
578
...
7090