Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................XX.....XXX..X..X] duration: 30.002391262s trace: https://jaeger.dev.storj.io/trace/3deeefe5cb22d838 FAIL (2026-03-18 21:47:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:33.016860025 (197h35m36.032529635s) [upload=189768]: Starting upload Mar 18 21:46:33.016989456 (197h35m36.032659062s) [upload=189768]: Done starting upload Mar 18 21:46:33.017052165 (197h35m36.03272177s) [upload=189768]: Uploading segments... Mar 18 21:46:33.017475356 (197h35m36.033144957s) [upload=189768]: Got next segment. Inline: false Mar 18 21:46:33.017539668 (197h35m36.03320927s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:33.017546318 (197h35m36.03321592s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:46:37.290282605 (197h35m40.305952212s) [seg_pos={0 0}, upload=189768]: Begin upload segment... Mar 18 21:46:37.290387539 (197h35m40.30605714s) [seg_pos={0 0}, upload=189768]: Done begin upload segment. Mar 18 21:46:37.290453240 (197h35m40.306122851s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.290923120 (197h35m40.30659273s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.291701810 (197h35m40.307371663s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292246979 (197h35m40.30791658s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292332991 (197h35m40.308002591s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292727864 (197h35m40.308397475s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292767367 (197h35m40.308436977s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293090621 (197h35m40.308760221s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293273873 (197h35m40.308943484s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293460336 (197h35m40.309129947s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293769369 (197h35m40.309438976s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293841511 (197h35m40.309511116s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294279896 (197h35m40.309949487s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294826369 (197h35m40.310495979s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295420929 (197h35m40.311090529s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295975630 (197h35m40.311645237s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.895634454 (197h35m40.911304055s) [upload=189768]: Next returned nil segment Mar 18 21:46:37.895651552 (197h35m40.911321163s) [upload=189768]: Waiting for error group managing segments... Mar 18 21:46:38.034797744 (197h35m41.05046737s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.034962121 (197h35m41.050631723s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.035419184 (197h35m41.051088784s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036603154 (197h35m41.052272755s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036707110 (197h35m41.052376712s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.037782694 (197h35m41.053452295s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039517974 (197h35m41.055187575s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039929620 (197h35m41.055599222s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040264640 (197h35m41.055934243s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040742968 (197h35m41.056412577s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.051123109 (197h35m41.066792713s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.056784301 (197h35m41.072453905s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082796373 (197h35m41.098465981s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082828521 (197h35m41.098498126s) [seg_pos={0 0}, upload=189768]: Satisfied cohort requirements after 13 pieces Mar 18 21:46:38.083138766 (197h35m41.098808369s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W4YnnqptDeH7t6QVMo988zAyAZ3Kj3dFE1yaxjttuWjrMvKeM): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083150948 (197h35m41.098820558s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083217334 (197h35m41.098886945s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083248682 (197h35m41.098918292s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083440350 (197h35m41.099109956s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083448598 (197h35m41.099118197s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083471664 (197h35m41.09914126s) [seg_pos={0 0}, upload=189768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:46:38.083578356 (197h35m41.099247957s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:46:38.083588526 (197h35m41.099258126s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:54.701340920 (197h35m57.717010532s) [upload=189768]: Done uploading segments... Mar 18 21:46:54.701366952 (197h35m57.717036561s) [upload=189768]: Upload finished. err: <nil> Duration: 30.002419338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX....XXXXXX..] duration: 21.708301964s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 21.708319921s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 32.267539216s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:39 UTC) orbiter warning: context deadline exceeded Duration: 32.267572524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X....X.XXXX.] duration: 26.620919711s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:46:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.621007176s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX..XX.] duration: 21.726106356s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:46:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 21.726220195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.002293216s trace: https://jaeger.dev.storj.io/trace/78d2ae33cae77162 FAIL (2026-03-18 21:46:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002342162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 47.828840268s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:06 UTC) orbiter warning: context deadline exceeded Duration: 47.828854707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XX....XXXXXX.] duration: 26.034862831s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 26.034895855s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XX.....XXX..X..] duration: 22.959785305s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:46:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 22.959816925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................X....X.XXXX] duration: 30.004684283s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:45:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00472724s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.00101487s trace: https://jaeger.dev.storj.io/trace/332ada5a7badd8c6 FAIL (2026-03-18 21:45:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001043189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.182926396s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:45:18 UTC) orbiter warning: context deadline exceeded Duration: 36.182954232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX.....XXX..X.] duration: 27.136646187s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 27.136676097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXX..XX] duration: 30.00119625s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 30.001237538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X....X.XXX] duration: 30.312073232s trace: https://jaeger.dev.storj.io/trace/318408f9135b4ebe FAIL (2026-03-18 21:44:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.312088323s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [................................................XX....XXXXXX] duration: 30.001095386s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:44:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001121943s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000879906s trace: https://jaeger.dev.storj.io/trace/29e3b11860b462ab FAIL (2026-03-18 21:44:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000920376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX.....XXX..X] duration: 30.003061155s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:44:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 30.003086874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X....X.XX] duration: 30.004686512s trace: https://jaeger.dev.storj.io/trace/741f8a58ad0c146 FAIL (2026-03-18 21:44:29 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004715111s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000849654s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:44:28 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00086881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX..X] duration: 30.001426519s trace: https://jaeger.dev.storj.io/trace/5aa167875471c0f5 FAIL (2026-03-18 21:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:43:50.615014606 (197h32m53.630684213s) [upload=189728]: Starting upload Mar 18 21:43:50.615152702 (197h32m53.630822309s) [upload=189728]: Done starting upload Mar 18 21:43:50.615302807 (197h32m53.630972394s) [upload=189728]: Uploading segments... Mar 18 21:43:50.615408894 (197h32m53.631078495s) [upload=189728]: Got next segment. Inline: true Mar 18 21:43:50.615417472 (197h32m53.631087079s) [upload=189728]: Waiting for error group managing segments... Mar 18 21:43:50.615502560 (197h32m53.63117217s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:43:50.615509793 (197h32m53.631179394s) [upload=189728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:43:50.615514503 (197h32m53.631184104s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:20.332317413 (197h33m23.347987024s) [upload=189728]: Done uploading segments... Mar 18 21:44:20.332388334 (197h33m23.348057949s) [upload=189728]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001466595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XX.....XXX..] duration: 22.966917102s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:44:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 22.966938471s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXX] duration: 5.003411737s trace: https://jaeger.dev.storj.io/trace/7ff31ea21f5ca509 FAIL (2026-03-18 21:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003428863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.....XXX.] duration: 23.296314259s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 23.296357598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXX] duration: 5.001107536s trace: https://jaeger.dev.storj.io/trace/88d5be3b8324a17 FAIL (2026-03-18 21:43:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX....XXXXX] duration: 30.001180974s trace: https://jaeger.dev.storj.io/trace/f1b7d8417d4adb6 FAIL (2026-03-18 21:43:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001193012s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....X.X] duration: 30.004475441s trace: https://jaeger.dev.storj.io/trace/3b96ae06fd84953d FAIL (2026-03-18 21:43:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004498817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 25.69905438s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 25.699084689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX....XXXX] duration: 30.001537599s trace: https://jaeger.dev.storj.io/trace/48ca5adafc30ca5f FAIL (2026-03-18 21:43:10 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001563183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XX.....XXX] duration: 30.0038269s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 30.003852084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 21.428238575s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 21.428257508s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXX] duration: 5.001548723s trace: https://jaeger.dev.storj.io/trace/66e6701df5ebc343 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001636972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 22.883333976s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.883365633s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001753067s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:42:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 30.001768186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX.....XX] duration: 30.00213144s trace: https://jaeger.dev.storj.io/trace/60e12ac05cfc6cad FAIL (2026-03-18 21:42:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:02.417656046 (197h31m5.433325652s) [upload=189697]: Starting upload Mar 18 21:42:02.417789276 (197h31m5.433458885s) [upload=189697]: Done starting upload Mar 18 21:42:02.418076625 (197h31m5.433746218s) [upload=189697]: Uploading segments... Mar 18 21:42:02.418408602 (197h31m5.434078206s) [upload=189697]: Got next segment. Inline: false Mar 18 21:42:02.418533265 (197h31m5.43420287s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:02.418540334 (197h31m5.434209936s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:18.695643535 (197h31m21.711313155s) [seg_pos={0 0}, upload=189697]: Begin upload segment... Mar 18 21:42:18.695691242 (197h31m21.711360841s) [seg_pos={0 0}, upload=189697]: Done begin upload segment. Mar 18 21:42:18.695716852 (197h31m21.711386465s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.695876691 (197h31m21.711546302s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696525961 (197h31m21.712195573s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696678593 (197h31m21.712348194s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696993890 (197h31m21.712663491s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697114858 (197h31m21.712784458s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697376505 (197h31m21.713046115s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697542371 (197h31m21.713211971s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697695110 (197h31m21.713364709s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698022818 (197h31m21.713692421s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698037811 (197h31m21.713707412s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698293088 (197h31m21.713962674s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698441877 (197h31m21.714111478s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698581463 (197h31m21.71425105s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698872122 (197h31m21.714541731s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.700720459 (197h31m21.716390053s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:19.287654268 (197h31m22.30332387s) [upload=189697]: Next returned nil segment Mar 18 21:42:19.287679198 (197h31m22.303348808s) [upload=189697]: Waiting for error group managing segments... Mar 18 21:42:19.424763134 (197h31m22.440432743s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.427461527 (197h31m22.44313113s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.432400301 (197h31m22.448069901s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436191214 (197h31m22.451860821s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436543389 (197h31m22.45221299s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.438064568 (197h31m22.453734168s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.441153829 (197h31m22.45682343s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445617792 (197h31m22.461287398s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445982446 (197h31m22.461652048s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.446667573 (197h31m22.46233718s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.449617438 (197h31m22.465287043s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.451584770 (197h31m22.467254371s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453078554 (197h31m22.468748158s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453096496 (197h31m22.468766097s) [seg_pos={0 0}, upload=189697]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:19.453377163 (197h31m22.469046772s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453384477 (197h31m22.46905408s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453527183 (197h31m22.46919679s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453533387 (197h31m22.469202987s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453677790 (197h31m22.469347399s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453690226 (197h31m22.469359826s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453711762 (197h31m22.469381354s) [seg_pos={0 0}, upload=189697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:19.453804025 (197h31m22.469473629s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:19.453812865 (197h31m22.46948248s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:27.988758516 (197h31m31.00442812s) [upload=189697]: Done uploading segments... Mar 18 21:42:27.988780210 (197h31m31.004449811s) [upload=189697]: Upload finished. err: <nil> Duration: 30.002157397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001367944s trace: https://jaeger.dev.storj.io/trace/2091b044248e599a FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:40.120881538 (197h30m43.136551138s) [upload=189693]: Starting upload Mar 18 21:41:40.121006332 (197h30m43.136675926s) [upload=189693]: Done starting upload Mar 18 21:41:40.121116258 (197h30m43.136785857s) [upload=189693]: Uploading segments... Mar 18 21:41:40.121231879 (197h30m43.136901483s) [upload=189693]: Got next segment. Inline: true Mar 18 21:41:40.121238612 (197h30m43.136908216s) [upload=189693]: Waiting for error group managing segments... Mar 18 21:41:40.121369082 (197h30m43.137038848s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:40.121377813 (197h30m43.137047422s) [upload=189693]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:40.121382320 (197h30m43.137051931s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:04.954963532 (197h31m7.970633133s) [upload=189693]: Done uploading segments... Mar 18 21:42:04.954984367 (197h31m7.970653961s) [upload=189693]: Upload finished. err: <nil> Duration:* 30.001393282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX....XXX] duration: 30.001088862s trace: https://jaeger.dev.storj.io/trace/1d7f86d24ec17772 FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001117797s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXX] duration: 5.000715173s trace: https://jaeger.dev.storj.io/trace/7736a03b25f2ff20 FAIL (2026-03-18 21:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000737863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.005395566s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005481469s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXX] duration: 5.000794843s trace: https://jaeger.dev.storj.io/trace/7a3bb9f117e697f7 FAIL (2026-03-18 21:41:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000818402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX.....X] duration: 30.003100164s trace: https://jaeger.dev.storj.io/trace/4fc9ef2ae31ecd22 FAIL (2026-03-18 21:41:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:15.503249526 (197h30m18.518919116s) [upload=189686]: Starting upload Mar 18 21:41:15.503318234 (197h30m18.518987825s) [upload=189686]: Done starting upload Mar 18 21:41:15.503408749 (197h30m18.519078341s) [upload=189686]: Uploading segments... Mar 18 21:41:15.503656555 (197h30m18.519326157s) [upload=189686]: Got next segment. Inline: false Mar 18 21:41:15.504053177 (197h30m18.519722786s) [upload=189686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:15.504067001 (197h30m18.519736612s) [upload=189686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:41:31.731468550 (197h30m34.747138153s) [seg_pos={0 0}, upload=189686]: Begin upload segment... Mar 18 21:41:31.731519809 (197h30m34.747189396s) [seg_pos={0 0}, upload=189686]: Done begin upload segment. Mar 18 21:41:31.731549457 (197h30m34.74721906s) [noise=true , address=145.79.227.15:4879 , piece_id=5HC4JLZPXTO6K75A2PFVRHATA3NGLCLHDHRUNDRHRX4GVGOB2YIA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.731587350 (197h30m34.74725697s) [noise=true , address=145.79.227.50:4374 , piece_id=TCPENHS7SPJAQ6ZX4JYFYOZZOGE4ZR2ATTMJCV26ISMLK7ZAY6UA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.732577179 (197h30m34.74824679s) [noise=true , address=45.91.116.7:4012 , piece_id=WLYVIBGTRJK7BSVGIBUZZ4X3N32WNRNK4VCOAZ5NA3KUVBNAVBRQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733071808 (197h30m34.748741415s) [noise=true , address=145.79.227.35:4481 , piece_id=G7TWEZJRZIZHNJAVFXYCGCP4OKVYBDHDQTM64TIIQGMLHVJUSBRQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733139183 (197h30m34.748808796s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733468284 (197h30m34.749137887s) [noise=true , address=145.79.227.5:4227 , piece_id=CTH3CMPJEYJDQETPHJRPIPDTBWORFHED525XIX2TKIHIVHGRUREQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733532881 (197h30m34.74920249s) [noise=true , address=145.79.227.38:4981 , piece_id=CLXD5PGJBIIWOZOUMB5XPWXUCLLTBOQQEWS6INMIV2F6ZKVZS65A, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733834983 (197h30m34.749504586s) [noise=true , address=145.79.227.8:4334 , piece_id=UB5BBUZD3F4O3A3E4MLEM2CYTPNFJTLGGFNMESULGHDQKLUXR4DA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734023775 (197h30m34.749693388s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734262300 (197h30m34.74993191s) [noise=true , address=145.79.227.14:4841 , piece_id=T5TWL2TDSDYDN3JG7PVWLYD4VNE55KEYFQHJ73G2FLBPXRAAYSZA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734417866 (197h30m34.750087453s) [noise=true , address=145.79.227.22:4511 , piece_id=7GXHYUPPUMRZU4ULJQN3AYHF4RLMVMNNCLO5GMZ2WJZT4V3N2DMA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734644759 (197h30m34.75031436s) [noise=true , address=145.79.227.23:4551 , piece_id=RJEPLWLVLFYHZWXVF3GZ2NMHEHS3VF7YDOPO5HHVHHK2Y3MQDVTQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734801466 (197h30m34.750471069s) [noise=true , address=145.79.227.16:4914 , piece_id=CRDK5SD6LSTTOXXORW2IIE5VSSD72ST5T2XLCNWELPJT6PM27JCQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735029769 (197h30m34.750699374s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735437298 (197h30m34.751106898s) [noise=true , address=77.90.31.12:4589 , piece_id=JURFT5XHNNZLX4VKC7D6F653KWZT47IJVWJDPYBBW34XJEVLOSGQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735877393 (197h30m34.751547003s) [noise=true , address=145.79.227.6:4260 , piece_id=DAZBN6ZO4CK55YAKDXFJOAFQVN3EYWSWD6RM3D3PDALXAWZWRITQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:32.333599746 (197h30m35.349269333s) [upload=189686]: Next returned nil segment Mar 18 21:41:32.333613110 (197h30m35.34928271s) [upload=189686]: Waiting for error group managing segments... Mar 18 21:41:32.459469423 (197h30m35.475139024s) [noise=true , address=77.90.31.12:4589 , piece_id=JURFT5XHNNZLX4VKC7D6F653KWZT47IJVWJDPYBBW34XJEVLOSGQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.469121284 (197h30m35.484790885s) [noise=true , address=145.79.227.50:4374 , piece_id=TCPENHS7SPJAQ6ZX4JYFYOZZOGE4ZR2ATTMJCV26ISMLK7ZAY6UA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.476940838 (197h30m35.492610439s) [noise=true , address=145.79.227.23:4551 , piece_id=RJEPLWLVLFYHZWXVF3GZ2NMHEHS3VF7YDOPO5HHVHHK2Y3MQDVTQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.479745098 (197h30m35.495414699s) [noise=true , address=145.79.227.8:4334 , piece_id=UB5BBUZD3F4O3A3E4MLEM2CYTPNFJTLGGFNMESULGHDQKLUXR4DA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.481099558 (197h30m35.49676916s) [noise=true , address=145.79.227.6:4260 , piece_id=DAZBN6ZO4CK55YAKDXFJOAFQVN3EYWSWD6RM3D3PDALXAWZWRITQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.482503144 (197h30m35.498172734s) [noise=true , address=145.79.227.16:4914 , piece_id=CRDK5SD6LSTTOXXORW2IIE5VSSD72ST5T2XLCNWELPJT6PM27JCQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.482906608 (197h30m35.498576209s) [noise=true , address=145.79.227.38:4981 , piece_id=CLXD5PGJBIIWOZOUMB5XPWXUCLLTBOQQEWS6INMIV2F6ZKVZS65A, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.483507975 (197h30m35.499177575s) [noise=true , address=145.79.227.22:4511 , piece_id=7GXHYUPPUMRZU4ULJQN3AYHF4RLMVMNNCLO5GMZ2WJZT4V3N2DMA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.485006092 (197h30m35.500675692s) [noise=true , address=145.79.227.5:4227 , piece_id=CTH3CMPJEYJDQETPHJRPIPDTBWORFHED525XIX2TKIHIVHGRUREQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.485595755 (197h30m35.501265356s) [noise=true , address=145.79.227.14:4841 , piece_id=T5TWL2TDSDYDN3JG7PVWLYD4VNE55KEYFQHJ73G2FLBPXRAAYSZA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.491892046 (197h30m35.507561646s) [noise=true , address=145.79.227.35:4481 , piece_id=G7TWEZJRZIZHNJAVFXYCGCP4OKVYBDHDQTM64TIIQGMLHVJUSBRQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492004041 (197h30m35.507673642s) [noise=true , address=145.79.227.15:4879 , piece_id=5HC4JLZPXTO6K75A2PFVRHATA3NGLCLHDHRUNDRHRX4GVGOB2YIA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492940751 (197h30m35.508610352s) [noise=true , address=45.91.116.7:4012 , piece_id=WLYVIBGTRJK7BSVGIBUZZ4X3N32WNRNK4VCOAZ5NA3KUVBNAVBRQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492965235 (197h30m35.508634839s) [seg_pos={0 0}, upload=189686]: Satisfied cohort requirements after 13 pieces Mar 18 21:41:32.493172636 (197h30m35.508842244s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493181025 (197h30m35.508850625s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493209444 (197h30m35.50887905s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LfpoDAb2seboZSoyGNp5zw2YZ4okmnvNZ5UQsgqgzn4MmDLJn): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493216579 (197h30m35.508886183s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493361946 (197h30m35.509031552s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493367612 (197h30m35.509037214s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493385412 (197h30m35.509054999s) [seg_pos={0 0}, upload=189686]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:41:32.493471565 (197h30m35.509141165s) [upload=189686]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:41:32.493476984 (197h30m35.509146584s) [upload=189686]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:41:41.004334124 (197h30m44.020003736s) [upload=189686]: Done uploading segments... Mar 18 21:41:41.004359380 (197h30m44.020028995s) [upload=189686]: Upload finished. err: <nil> Duration: 30.003123553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX....XX] duration: 30.001888276s trace: https://jaeger.dev.storj.io/trace/a150b95b00beb34 FAIL (2026-03-18 21:41:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001927046s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXX] duration: 5.000911274s trace: https://jaeger.dev.storj.io/trace/52716293d774ae67 FAIL (2026-03-18 21:41:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000938911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001494239s trace: https://jaeger.dev.storj.io/trace/4ac054bffad9608 FAIL (2026-03-18 21:40:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:40:26.840889435 (197h29m29.856559038s) [upload=189674]: Starting upload Mar 18 21:40:26.840962647 (197h29m29.856632239s) [upload=189674]: Done starting upload Mar 18 21:40:26.841064766 (197h29m29.856734359s) [upload=189674]: Uploading segments... Mar 18 21:40:26.841123788 (197h29m29.856793395s) [upload=189674]: Got next segment. Inline: true Mar 18 21:40:26.841129712 (197h29m29.856799316s) [upload=189674]: Waiting for error group managing segments... Mar 18 21:40:26.841184594 (197h29m29.856854194s) [upload=189674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:40:26.841190017 (197h29m29.856859618s) [upload=189674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:40:26.841193766 (197h29m29.856863358s) [upload=189674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:40:43.578971735 (197h29m46.594641345s) [upload=189674]: Done uploading segments... Mar 18 21:40:43.579007655 (197h29m46.594677257s) [upload=189674]: Upload finished. err: <nil> Duration:* 30.001515174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 23.105496299s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:40:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 23.105528533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 22.903290305s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:40:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.903363153s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XX] duration: 5.000689315s trace: https://jaeger.dev.storj.io/trace/7671635b36f95e9a FAIL (2026-03-18 21:40:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000717963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XX....X] duration: 30.001483491s trace: https://jaeger.dev.storj.io/trace/1dd5f2bde0ec2260 FAIL (2026-03-18 21:40:39 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00150831s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 23.819500544s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:40:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 23.81953123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 12.5256823s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:39:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 12.525711558s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)