Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 2.29321787s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:52:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293234089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.700045106s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:51:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.700069491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 3.079148452s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.079181132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.627356513s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:51:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.627384059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.670302209s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:50:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.670371107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.092866709s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:50:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.092911791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.614631072s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:50:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.615454333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.343241231s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:49:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.343637661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.649074906s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:49:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649099151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.078109435s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:49:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.078138539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.221499652s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:48:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.221519553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.830031145s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:48:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.830058141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.630820659s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:48:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.630851475s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.629025861s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:47:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.629050657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.78988013s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:47:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.789900947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 1.656634785s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:47:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.656656781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.193214873s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:47:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.193242496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.627382348s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.627425977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.82647465s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:46:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.826498869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 1.600394987s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:46:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.600423057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.387361096s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:46:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.387410921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.173282695s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:46:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.173310599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.094215812s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.094238525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 1.713760816s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:45:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.713796912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.842473207s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.842561141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.100590084s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:44:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.100618386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 1.726942295s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:44:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.727060924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.969321084s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.969371748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.977720381s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:43:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.977751356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.295915949s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:43:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.295944648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 1.938541083s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.938580433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.714899069s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:43:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.714925396s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 1.841347468s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:43:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.841395642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.118197122s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:43:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.118236859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.819018119s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:42:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819066001s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.841961137s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:42:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.84201072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 1.700699569s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:42:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.700724695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.070731301s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.070757393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.388069643s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:41:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388095657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.107072841s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:41:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.10709592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 1.658160094s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:41:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658196615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 1.667563657s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:40:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.667585558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.916826936s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:40:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.916844415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.332784851s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:40:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.332816748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.063556924s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.063579517s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 1.698329287s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:40:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.698342311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.767751797s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:40:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76777809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.272315794s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:39:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.272372518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 1.643928158s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:39:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.64395414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.883098047s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:39:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.883118176s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)