Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.002030887s trace: https://jaeger.dev.storj.io/trace/593ce046ea9a8247 FAIL (2026-03-18 21:48:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002066647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XXX..XX.XX] duration: 30.002564875s trace: https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5 FAIL (2026-03-18 21:48:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil> Duration:* 30.002590336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [............................................XX....XXXXXX..XX] duration: 30.001857974s trace: https://jaeger.dev.storj.io/trace/6dbd9885f9de107d FAIL (2026-03-18 21:48:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001872098s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................X....X.XXXX.XX] duration: 30.116176448s trace: https://jaeger.dev.storj.io/trace/2bc345e63b93f9dd FAIL (2026-03-18 21:48:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.116205418s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 51.36761058s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:48:09 UTC) orbiter warning: context deadline exceeded Duration: 51.367641032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XX.....XXX..X..XX] duration: 30.003642955s trace: https://jaeger.dev.storj.io/trace/44d224acaa004e0 FAIL (2026-03-18 21:47:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:18.442808833 (197h36m21.458478442s) [upload=189779]: Starting upload Mar 18 21:47:18.442901121 (197h36m21.458570723s) [upload=189779]: Done starting upload Mar 18 21:47:18.442976117 (197h36m21.458645717s) [upload=189779]: Uploading segments... Mar 18 21:47:18.443606581 (197h36m21.459276181s) [upload=189779]: Got next segment. Inline: false Mar 18 21:47:18.443680474 (197h36m21.459350077s) [upload=189779]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:18.443699000 (197h36m21.459368603s) [upload=189779]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:47:26.699422791 (197h36m29.715092406s) [seg_pos={0 0}, upload=189779]: Begin upload segment... Mar 18 21:47:26.699491248 (197h36m29.715160841s) [seg_pos={0 0}, upload=189779]: Done begin upload segment. Mar 18 21:47:26.699526331 (197h36m29.715195943s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702077766 (197h36m29.717747375s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702861330 (197h36m29.71853093s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702891630 (197h36m29.718561232s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703323314 (197h36m29.718992924s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703352656 (197h36m29.71902226s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703987436 (197h36m29.719657047s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703997584 (197h36m29.719667195s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.704537437 (197h36m29.720207043s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.704977460 (197h36m29.720647062s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.706372766 (197h36m29.722042367s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.707008899 (197h36m29.722678497s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.707089684 (197h36m29.722759294s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.708051043 (197h36m29.723720636s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.709186798 (197h36m29.724856393s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.712310630 (197h36m29.727980233s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:27.306615628 (197h36m30.322285255s) [upload=189779]: Next returned nil segment Mar 18 21:47:27.306668889 (197h36m30.322338504s) [upload=189779]: Waiting for error group managing segments... Mar 18 21:47:27.442419844 (197h36m30.458089445s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.442434040 (197h36m30.458103645s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.444434343 (197h36m30.460103947s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.445531395 (197h36m30.46120101s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.445728504 (197h36m30.461398114s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.448910049 (197h36m30.46457967s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.451992678 (197h36m30.467662287s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.452128814 (197h36m30.467798432s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.454127332 (197h36m30.469796945s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.455872225 (197h36m30.471541839s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.458476641 (197h36m30.474146241s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.461198518 (197h36m30.47686815s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.466630469 (197h36m30.482300072s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.466653132 (197h36m30.482322737s) [seg_pos={0 0}, upload=189779]: Satisfied cohort requirements after 13 pieces Mar 18 21:47:27.466926351 (197h36m30.482595955s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.466933928 (197h36m30.482603527s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.466972928 (197h36m30.482642554s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.466988039 (197h36m30.482657651s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.467169807 (197h36m30.482839418s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.467179258 (197h36m30.482848868s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.467211812 (197h36m30.482881442s) [seg_pos={0 0}, upload=189779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:47:27.467312886 (197h36m30.482982494s) [upload=189779]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:47:27.467320096 (197h36m30.482989711s) [upload=189779]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:35.897519767 (197h36m38.913189375s) [upload=189779]: Done uploading segments... Mar 18 21:47:35.897549016 (197h36m38.91321862s) [upload=189779]: Upload finished. err: <nil> Duration: 30.003669693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.000666722s trace: https://jaeger.dev.storj.io/trace/549c36531950e590 FAIL (2026-03-18 21:47:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000823006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X....X.XXXX.X] duration: 30.004707963s trace: https://jaeger.dev.storj.io/trace/1799e0deebe9782a FAIL (2026-03-18 21:47:27 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004736265s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XX....XXXXXX..X] duration: 30.001842368s trace: https://jaeger.dev.storj.io/trace/f51217a9a46a2f1 FAIL (2026-03-18 21:47:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001856253s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XXX..XX.X] duration: 30.001896226s trace: https://jaeger.dev.storj.io/trace/79f59160b77ec97a FAIL (2026-03-18 21:47:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:44.213562861 (197h35m47.229232461s) [upload=189770]: Starting upload Mar 18 21:46:44.213634624 (197h35m47.229304213s) [upload=189770]: Done starting upload Mar 18 21:46:44.213688107 (197h35m47.229357718s) [upload=189770]: Uploading segments... Mar 18 21:46:44.213783933 (197h35m47.229453544s) [upload=189770]: Got next segment. Inline: true Mar 18 21:46:44.213793186 (197h35m47.229462797s) [upload=189770]: Waiting for error group managing segments... Mar 18 21:46:44.213873893 (197h35m47.229543493s) [upload=189770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:44.213881750 (197h35m47.229551356s) [upload=189770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:46:44.213887721 (197h35m47.229557323s) [upload=189770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:56.859337880 (197h35m59.875007483s) [upload=189770]: Done uploading segments... Mar 18 21:46:56.859357249 (197h35m59.875026849s) [upload=189770]: Upload finished. err: <nil> Duration:* 30.001921231s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.001462524s trace: https://jaeger.dev.storj.io/trace/6f36e79e78a01382 FAIL (2026-03-18 21:47:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001491533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................XX.....XXX..X..X] duration: 30.002391262s trace: https://jaeger.dev.storj.io/trace/3deeefe5cb22d838 FAIL (2026-03-18 21:47:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:33.016860025 (197h35m36.032529635s) [upload=189768]: Starting upload Mar 18 21:46:33.016989456 (197h35m36.032659062s) [upload=189768]: Done starting upload Mar 18 21:46:33.017052165 (197h35m36.03272177s) [upload=189768]: Uploading segments... Mar 18 21:46:33.017475356 (197h35m36.033144957s) [upload=189768]: Got next segment. Inline: false Mar 18 21:46:33.017539668 (197h35m36.03320927s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:33.017546318 (197h35m36.03321592s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:46:37.290282605 (197h35m40.305952212s) [seg_pos={0 0}, upload=189768]: Begin upload segment... Mar 18 21:46:37.290387539 (197h35m40.30605714s) [seg_pos={0 0}, upload=189768]: Done begin upload segment. Mar 18 21:46:37.290453240 (197h35m40.306122851s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.290923120 (197h35m40.30659273s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.291701810 (197h35m40.307371663s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292246979 (197h35m40.30791658s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292332991 (197h35m40.308002591s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292727864 (197h35m40.308397475s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292767367 (197h35m40.308436977s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293090621 (197h35m40.308760221s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293273873 (197h35m40.308943484s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293460336 (197h35m40.309129947s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293769369 (197h35m40.309438976s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293841511 (197h35m40.309511116s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294279896 (197h35m40.309949487s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294826369 (197h35m40.310495979s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295420929 (197h35m40.311090529s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295975630 (197h35m40.311645237s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.895634454 (197h35m40.911304055s) [upload=189768]: Next returned nil segment Mar 18 21:46:37.895651552 (197h35m40.911321163s) [upload=189768]: Waiting for error group managing segments... Mar 18 21:46:38.034797744 (197h35m41.05046737s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.034962121 (197h35m41.050631723s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.035419184 (197h35m41.051088784s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036603154 (197h35m41.052272755s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036707110 (197h35m41.052376712s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.037782694 (197h35m41.053452295s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039517974 (197h35m41.055187575s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039929620 (197h35m41.055599222s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040264640 (197h35m41.055934243s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040742968 (197h35m41.056412577s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.051123109 (197h35m41.066792713s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.056784301 (197h35m41.072453905s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082796373 (197h35m41.098465981s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082828521 (197h35m41.098498126s) [seg_pos={0 0}, upload=189768]: Satisfied cohort requirements after 13 pieces Mar 18 21:46:38.083138766 (197h35m41.098808369s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W4YnnqptDeH7t6QVMo988zAyAZ3Kj3dFE1yaxjttuWjrMvKeM): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083150948 (197h35m41.098820558s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083217334 (197h35m41.098886945s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083248682 (197h35m41.098918292s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083440350 (197h35m41.099109956s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083448598 (197h35m41.099118197s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083471664 (197h35m41.09914126s) [seg_pos={0 0}, upload=189768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:46:38.083578356 (197h35m41.099247957s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:46:38.083588526 (197h35m41.099258126s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:54.701340920 (197h35m57.717010532s) [upload=189768]: Done uploading segments... Mar 18 21:46:54.701366952 (197h35m57.717036561s) [upload=189768]: Upload finished. err: <nil> Duration: 30.002419338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX....XXXXXX..] duration: 21.708301964s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 21.708319921s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 32.267539216s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:39 UTC) orbiter warning: context deadline exceeded Duration: 32.267572524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X....X.XXXX.] duration: 26.620919711s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:46:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.621007176s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX..XX.] duration: 21.726106356s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:46:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 21.726220195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.002293216s trace: https://jaeger.dev.storj.io/trace/78d2ae33cae77162 FAIL (2026-03-18 21:46:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002342162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 47.828840268s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:06 UTC) orbiter warning: context deadline exceeded Duration: 47.828854707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XX....XXXXXX.] duration: 26.034862831s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 26.034895855s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XX.....XXX..X..] duration: 22.959785305s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:46:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 22.959816925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................X....X.XXXX] duration: 30.004684283s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:45:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00472724s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.00101487s trace: https://jaeger.dev.storj.io/trace/332ada5a7badd8c6 FAIL (2026-03-18 21:45:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001043189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.182926396s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:45:18 UTC) orbiter warning: context deadline exceeded Duration: 36.182954232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX.....XXX..X.] duration: 27.136646187s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 27.136676097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXX..XX] duration: 30.00119625s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 30.001237538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X....X.XXX] duration: 30.312073232s trace: https://jaeger.dev.storj.io/trace/318408f9135b4ebe FAIL (2026-03-18 21:44:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.312088323s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [................................................XX....XXXXXX] duration: 30.001095386s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:44:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001121943s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000879906s trace: https://jaeger.dev.storj.io/trace/29e3b11860b462ab FAIL (2026-03-18 21:44:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000920376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX.....XXX..X] duration: 30.003061155s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:44:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 30.003086874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X....X.XX] duration: 30.004686512s trace: https://jaeger.dev.storj.io/trace/741f8a58ad0c146 FAIL (2026-03-18 21:44:29 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004715111s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000849654s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:44:28 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00086881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX..X] duration: 30.001426519s trace: https://jaeger.dev.storj.io/trace/5aa167875471c0f5 FAIL (2026-03-18 21:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:43:50.615014606 (197h32m53.630684213s) [upload=189728]: Starting upload Mar 18 21:43:50.615152702 (197h32m53.630822309s) [upload=189728]: Done starting upload Mar 18 21:43:50.615302807 (197h32m53.630972394s) [upload=189728]: Uploading segments... Mar 18 21:43:50.615408894 (197h32m53.631078495s) [upload=189728]: Got next segment. Inline: true Mar 18 21:43:50.615417472 (197h32m53.631087079s) [upload=189728]: Waiting for error group managing segments... Mar 18 21:43:50.615502560 (197h32m53.63117217s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:43:50.615509793 (197h32m53.631179394s) [upload=189728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:43:50.615514503 (197h32m53.631184104s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:20.332317413 (197h33m23.347987024s) [upload=189728]: Done uploading segments... Mar 18 21:44:20.332388334 (197h33m23.348057949s) [upload=189728]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001466595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XX.....XXX..] duration: 22.966917102s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:44:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 22.966938471s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXX] duration: 5.003411737s trace: https://jaeger.dev.storj.io/trace/7ff31ea21f5ca509 FAIL (2026-03-18 21:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003428863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.....XXX.] duration: 23.296314259s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 23.296357598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXX] duration: 5.001107536s trace: https://jaeger.dev.storj.io/trace/88d5be3b8324a17 FAIL (2026-03-18 21:43:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX....XXXXX] duration: 30.001180974s trace: https://jaeger.dev.storj.io/trace/f1b7d8417d4adb6 FAIL (2026-03-18 21:43:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001193012s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....X.X] duration: 30.004475441s trace: https://jaeger.dev.storj.io/trace/3b96ae06fd84953d FAIL (2026-03-18 21:43:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004498817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 25.69905438s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 25.699084689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX....XXXX] duration: 30.001537599s trace: https://jaeger.dev.storj.io/trace/48ca5adafc30ca5f FAIL (2026-03-18 21:43:10 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001563183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XX.....XXX] duration: 30.0038269s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 30.003852084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 21.428238575s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 21.428257508s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXX] duration: 5.001548723s trace: https://jaeger.dev.storj.io/trace/66e6701df5ebc343 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001636972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 22.883333976s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.883365633s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001753067s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:42:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 30.001768186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX.....XX] duration: 30.00213144s trace: https://jaeger.dev.storj.io/trace/60e12ac05cfc6cad FAIL (2026-03-18 21:42:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:02.417656046 (197h31m5.433325652s) [upload=189697]: Starting upload Mar 18 21:42:02.417789276 (197h31m5.433458885s) [upload=189697]: Done starting upload Mar 18 21:42:02.418076625 (197h31m5.433746218s) [upload=189697]: Uploading segments... Mar 18 21:42:02.418408602 (197h31m5.434078206s) [upload=189697]: Got next segment. Inline: false Mar 18 21:42:02.418533265 (197h31m5.43420287s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:02.418540334 (197h31m5.434209936s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:18.695643535 (197h31m21.711313155s) [seg_pos={0 0}, upload=189697]: Begin upload segment... Mar 18 21:42:18.695691242 (197h31m21.711360841s) [seg_pos={0 0}, upload=189697]: Done begin upload segment. Mar 18 21:42:18.695716852 (197h31m21.711386465s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.695876691 (197h31m21.711546302s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696525961 (197h31m21.712195573s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696678593 (197h31m21.712348194s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696993890 (197h31m21.712663491s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697114858 (197h31m21.712784458s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697376505 (197h31m21.713046115s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697542371 (197h31m21.713211971s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697695110 (197h31m21.713364709s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698022818 (197h31m21.713692421s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698037811 (197h31m21.713707412s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698293088 (197h31m21.713962674s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698441877 (197h31m21.714111478s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698581463 (197h31m21.71425105s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698872122 (197h31m21.714541731s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.700720459 (197h31m21.716390053s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:19.287654268 (197h31m22.30332387s) [upload=189697]: Next returned nil segment Mar 18 21:42:19.287679198 (197h31m22.303348808s) [upload=189697]: Waiting for error group managing segments... Mar 18 21:42:19.424763134 (197h31m22.440432743s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.427461527 (197h31m22.44313113s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.432400301 (197h31m22.448069901s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436191214 (197h31m22.451860821s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436543389 (197h31m22.45221299s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.438064568 (197h31m22.453734168s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.441153829 (197h31m22.45682343s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445617792 (197h31m22.461287398s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445982446 (197h31m22.461652048s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.446667573 (197h31m22.46233718s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.449617438 (197h31m22.465287043s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.451584770 (197h31m22.467254371s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453078554 (197h31m22.468748158s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453096496 (197h31m22.468766097s) [seg_pos={0 0}, upload=189697]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:19.453377163 (197h31m22.469046772s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453384477 (197h31m22.46905408s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453527183 (197h31m22.46919679s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453533387 (197h31m22.469202987s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453677790 (197h31m22.469347399s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453690226 (197h31m22.469359826s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453711762 (197h31m22.469381354s) [seg_pos={0 0}, upload=189697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:19.453804025 (197h31m22.469473629s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:19.453812865 (197h31m22.46948248s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:27.988758516 (197h31m31.00442812s) [upload=189697]: Done uploading segments... Mar 18 21:42:27.988780210 (197h31m31.004449811s) [upload=189697]: Upload finished. err: <nil> Duration: 30.002157397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001367944s trace: https://jaeger.dev.storj.io/trace/2091b044248e599a FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:40.120881538 (197h30m43.136551138s) [upload=189693]: Starting upload Mar 18 21:41:40.121006332 (197h30m43.136675926s) [upload=189693]: Done starting upload Mar 18 21:41:40.121116258 (197h30m43.136785857s) [upload=189693]: Uploading segments... Mar 18 21:41:40.121231879 (197h30m43.136901483s) [upload=189693]: Got next segment. Inline: true Mar 18 21:41:40.121238612 (197h30m43.136908216s) [upload=189693]: Waiting for error group managing segments... Mar 18 21:41:40.121369082 (197h30m43.137038848s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:40.121377813 (197h30m43.137047422s) [upload=189693]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:40.121382320 (197h30m43.137051931s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:04.954963532 (197h31m7.970633133s) [upload=189693]: Done uploading segments... Mar 18 21:42:04.954984367 (197h31m7.970653961s) [upload=189693]: Upload finished. err: <nil> Duration:* 30.001393282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX....XXX] duration: 30.001088862s trace: https://jaeger.dev.storj.io/trace/1d7f86d24ec17772 FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001117797s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXX] duration: 5.000715173s trace: https://jaeger.dev.storj.io/trace/7736a03b25f2ff20 FAIL (2026-03-18 21:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000737863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.005395566s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005481469s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)