Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 48.089832724s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:50:43 UTC) orbiter warning: context deadline exceeded Duration: 48.089861146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........................................X....X.XXXX.XXX..] duration: 22.944071055s trace: https://jaeger.dev.storj.io/trace/11dc3d42043d3b39 FAIL (2026-03-18 21:50:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.944089644s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.001467783s trace: https://jaeger.dev.storj.io/trace/6d606ff3c1dec74d FAIL (2026-03-18 21:50:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0014965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........................................XX.....XXX..X..XXX.X] duration: 30.002587748s trace: https://jaeger.dev.storj.io/trace/40000dbead1c6ff1 FAIL (2026-03-18 21:50:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:49:54.729674692 (197h38m57.745344303s) [upload=189820]: Starting upload Mar 18 21:49:54.729779028 (197h38m57.745448654s) [upload=189820]: Done starting upload Mar 18 21:49:54.729884587 (197h38m57.745554178s) [upload=189820]: Uploading segments... Mar 18 21:49:54.730169512 (197h38m57.745839117s) [upload=189820]: Got next segment. Inline: false Mar 18 21:49:54.730212358 (197h38m57.745881959s) [upload=189820]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:49:54.730219331 (197h38m57.745888932s) [upload=189820]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:49:59.000941337 (197h39m2.016610947s) [seg_pos={0 0}, upload=189820]: Begin upload segment... Mar 18 21:49:59.001024661 (197h39m2.01669427s) [seg_pos={0 0}, upload=189820]: Done begin upload segment. Mar 18 21:49:59.001139181 (197h39m2.016808792s) [noise=true , address=145.79.227.46:4605 , piece_id=6TEEMJ5ICGSJHWLUDWEEEUIFZLMTVATY4WJJRXKB3SMM3BQT7GZQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.002137662 (197h39m2.017807296s) [noise=true , address=145.79.227.22:4514 , piece_id=VZRJUNCLCCYNERWWVWMH2VQELGGEPBRMENDUBJSS54WTQIFTW5IA, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.003032612 (197h39m2.018702223s) [noise=true , address=145.79.227.17:4974 , piece_id=MSDS6FXLSKG4GPLEI6GN5XJGSZM3MJNWWLAL6UXUHO5N3NLKWGGQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.003594226 (197h39m2.019263836s) [noise=true , address=145.79.227.13:4806 , piece_id=NCS4MEL4HILNFCKMJYOOWMAH2Z4KBVMT2QYAOV5YDLEAFFMTWI2A, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.004361981 (197h39m2.020031587s) [noise=true , address=145.79.227.26:4657 , piece_id=X2F64ILVCS2HHDHCQHJQWV6TLEWHNJLNJBEMKULI5VOODLAEZ5NQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.004846847 (197h39m2.02051645s) [noise=true , address=145.79.227.5:4224 , piece_id=7H2SMD77J74GBV6V6TTGE4CCWCXRH64K5IEGXYF4YWMM3GHW5ZWQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.005292749 (197h39m2.020962356s) [noise=true , address=145.79.227.49:4338 , piece_id=Q7Y76CB3PHC3W62X3WNGJSSDGB4ODNTVX4N2VG4P26LOCCFFDPEA, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.005695681 (197h39m2.021365273s) [noise=true , address=145.79.227.23:4548 , piece_id=4CLWTCX6FHDGTGIDII3CTMTIZTAL7CJFMLGOYEXQNM7KMBIBJUSQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.007974271 (197h39m2.023643881s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.008691613 (197h39m2.024361222s) [noise=true , address=145.79.227.16:4915 , piece_id=TE7XEJRRETUJ5I4VVMKP2J7F3ZPTY2NWNTHNJLXSBXZF3UOTZKQA, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.010263764 (197h39m2.025933374s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.010892680 (197h39m2.02656229s) [noise=true , address=145.79.227.50:4375 , piece_id=XSRUK2PNASDBASH56XTL2GXXWVTDUFRHZCFXZHCBXGUJ6ZAM4JFQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.011500081 (197h39m2.027169693s) [noise=true , address=45.91.116.4:4939 , piece_id=NRM24OUWBKMLXL3K3JEUXHAK74YXZZ3HWIDUF26HKWANGMWPRAZA, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.013458920 (197h39m2.029128548s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.015443018 (197h39m2.031112629s) [noise=true , address=145.79.227.33:4870 , piece_id=OO4QWSZQR4DLX34TBT7MH7MGUYQVXPS4GU4QI5GN4OPBIHLPC3ZQ, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.017622563 (197h39m2.03329217s) [noise=true , address=45.91.116.12:4227 , piece_id=NZNDE5Y4BGU7OYHAXJU6TMNF2LM3PE4GJNKORAWRBY4HZFUKAP4A, seg_pos={0 0}, upload=189820]: Uploading piece... Mar 18 21:49:59.615985686 (197h39m2.631655279s) [upload=189820]: Next returned nil segment Mar 18 21:49:59.615999572 (197h39m2.631669176s) [upload=189820]: Waiting for error group managing segments... Mar 18 21:49:59.748555653 (197h39m2.764225253s) [noise=true , address=145.79.227.17:4974 , piece_id=MSDS6FXLSKG4GPLEI6GN5XJGSZM3MJNWWLAL6UXUHO5N3NLKWGGQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.750995663 (197h39m2.766665263s) [noise=true , address=145.79.227.23:4548 , piece_id=4CLWTCX6FHDGTGIDII3CTMTIZTAL7CJFMLGOYEXQNM7KMBIBJUSQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.754590755 (197h39m2.770260357s) [noise=true , address=145.79.227.16:4915 , piece_id=TE7XEJRRETUJ5I4VVMKP2J7F3ZPTY2NWNTHNJLXSBXZF3UOTZKQA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.757716103 (197h39m2.773385704s) [noise=true , address=145.79.227.22:4514 , piece_id=VZRJUNCLCCYNERWWVWMH2VQELGGEPBRMENDUBJSS54WTQIFTW5IA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.757759155 (197h39m2.773428741s) [noise=true , address=145.79.227.5:4224 , piece_id=7H2SMD77J74GBV6V6TTGE4CCWCXRH64K5IEGXYF4YWMM3GHW5ZWQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.760731165 (197h39m2.776400783s) [noise=true , address=145.79.227.13:4806 , piece_id=NCS4MEL4HILNFCKMJYOOWMAH2Z4KBVMT2QYAOV5YDLEAFFMTWI2A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.761241065 (197h39m2.776910666s) [noise=true , address=145.79.227.26:4657 , piece_id=X2F64ILVCS2HHDHCQHJQWV6TLEWHNJLNJBEMKULI5VOODLAEZ5NQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.762447085 (197h39m2.778116677s) [noise=true , address=145.79.227.46:4605 , piece_id=6TEEMJ5ICGSJHWLUDWEEEUIFZLMTVATY4WJJRXKB3SMM3BQT7GZQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.763006585 (197h39m2.778676215s) [noise=true , address=145.79.227.50:4375 , piece_id=XSRUK2PNASDBASH56XTL2GXXWVTDUFRHZCFXZHCBXGUJ6ZAM4JFQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.763184247 (197h39m2.778853848s) [noise=true , address=45.91.116.12:4227 , piece_id=NZNDE5Y4BGU7OYHAXJU6TMNF2LM3PE4GJNKORAWRBY4HZFUKAP4A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.767249209 (197h39m2.782918826s) [noise=true , address=45.91.116.4:4939 , piece_id=NRM24OUWBKMLXL3K3JEUXHAK74YXZZ3HWIDUF26HKWANGMWPRAZA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.768646318 (197h39m2.78431592s) [noise=true , address=145.79.227.49:4338 , piece_id=Q7Y76CB3PHC3W62X3WNGJSSDGB4ODNTVX4N2VG4P26LOCCFFDPEA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.769381823 (197h39m2.785051415s) [noise=true , address=145.79.227.33:4870 , piece_id=OO4QWSZQR4DLX34TBT7MH7MGUYQVXPS4GU4QI5GN4OPBIHLPC3ZQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil> Mar 18 21:49:59.769397445 (197h39m2.785067046s) [seg_pos={0 0}, upload=189820]: Satisfied cohort requirements after 13 pieces Mar 18 21:49:59.769620863 (197h39m2.78529047s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:59.769627950 (197h39m2.785297551s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold Mar 18 21:49:59.770067206 (197h39m2.785736816s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:59.770078138 (197h39m2.785747742s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold Mar 18 21:49:59.770220142 (197h39m2.785889773s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF Mar 18 21:49:59.770233071 (197h39m2.785902684s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold Mar 18 21:49:59.770262306 (197h39m2.785931893s) [seg_pos={0 0}, upload=189820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:49:59.770377827 (197h39m2.786047438s) [upload=189820]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:49:59.770385336 (197h39m2.786054946s) [upload=189820]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:50:20.065891921 (197h39m23.081561528s) [upload=189820]: Done uploading segments... Mar 18 21:50:20.065970376 (197h39m23.081639982s) [upload=189820]: Upload finished. err: <nil> Duration: 30.002606203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............................................X....X.XXXX.XXX.] duration: 26.812748906s trace: https://jaeger.dev.storj.io/trace/11dc3d42043d3b39 FAIL (2026-03-18 21:50:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.812781072s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [..........................................XX....XXXXXX..XX.X] duration: 30.003331458s trace: https://jaeger.dev.storj.io/trace/22e91f00228cb631 FAIL (2026-03-18 21:50:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.003352273s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [................................................XXX..XX.XX..] duration: 29.846653241s trace: https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5 FAIL (2026-03-18 21:49:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil> Duration:* 29.846692469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................................XX.....XXX..X..XXX.] duration: 27.19908997s trace: https://jaeger.dev.storj.io/trace/251f894af766eb83 FAIL (2026-03-18 21:49:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:48:32.645171629 (197h37m35.660841237s) [upload=189796]: Starting upload Mar 18 21:48:32.645326878 (197h37m35.660996481s) [upload=189796]: Done starting upload Mar 18 21:48:32.645418130 (197h37m35.661087732s) [upload=189796]: Uploading segments... Mar 18 21:48:32.645810289 (197h37m35.661479903s) [upload=189796]: Got next segment. Inline: false Mar 18 21:48:32.645974442 (197h37m35.661644261s) [upload=189796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:48:32.645981851 (197h37m35.661651455s) [upload=189796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:48:45.066247463 (197h37m48.081917068s) [seg_pos={0 0}, upload=189796]: Begin upload segment... Mar 18 21:48:45.066313671 (197h37m48.081983272s) [seg_pos={0 0}, upload=189796]: Done begin upload segment. Mar 18 21:48:45.066379707 (197h37m48.082049312s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.066410147 (197h37m48.08207974s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.066971840 (197h37m48.082641444s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.067083170 (197h37m48.082752778s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.067554140 (197h37m48.083223754s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.068499259 (197h37m48.084168865s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.068748780 (197h37m48.084418385s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.069466321 (197h37m48.085135933s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.069654213 (197h37m48.085323824s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.070484066 (197h37m48.086153671s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.071245408 (197h37m48.086915016s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.071594691 (197h37m48.087264302s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.072277538 (197h37m48.087947139s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074170774 (197h37m48.08984036s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074261023 (197h37m48.089930611s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074812150 (197h37m48.090481742s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.668879318 (197h37m48.684548905s) [upload=189796]: Next returned nil segment Mar 18 21:48:45.668896432 (197h37m48.684566039s) [upload=189796]: Waiting for error group managing segments... Mar 18 21:48:45.797763739 (197h37m48.81343336s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.805326384 (197h37m48.820995984s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.806217578 (197h37m48.821887189s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.809363726 (197h37m48.825033341s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.811971344 (197h37m48.827640944s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.812960498 (197h37m48.828630103s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.813854402 (197h37m48.829524002s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.814801800 (197h37m48.830471401s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.823306084 (197h37m48.838975702s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.823905178 (197h37m48.839574778s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.824610241 (197h37m48.840279827s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.824966367 (197h37m48.840635966s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.831671478 (197h37m48.847341091s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.831697992 (197h37m48.847367609s) [seg_pos={0 0}, upload=189796]: Satisfied cohort requirements after 13 pieces Mar 18 21:48:45.831975091 (197h37m48.8476447s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pvLo4F5ieDtepopg8Wi8jw6PKNeDCDvtH5iVNQXBUQwSXYxuD): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.831986640 (197h37m48.847656251s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837502196 (197h37m48.85317181s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.837522368 (197h37m48.853191978s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837711338 (197h37m48.853380951s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.837720674 (197h37m48.853390286s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837745572 (197h37m48.853415159s) [seg_pos={0 0}, upload=189796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:48:45.837833743 (197h37m48.853503354s) [upload=189796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:48:45.837839430 (197h37m48.853509038s) [upload=189796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:54.383402636 (197h37m57.399072251s) [upload=189796]: Done uploading segments... Mar 18 21:48:54.383428351 (197h37m57.399097962s) [upload=189796]: Upload finished. err: <nil> Duration: 27.199114115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 27.770362689s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:49:52 UTC) orbiter warning: context deadline exceeded Duration: 27.770379656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.00094823s trace: https://jaeger.dev.storj.io/trace/2a1ec0d7f3bca161 FAIL (2026-03-18 21:49:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000971822s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 40.110626321s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:49:25 UTC) orbiter warning: context deadline exceeded Duration: 40.110646606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XXX..XX.XX.] duration: 21.787423914s trace: https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5 FAIL (2026-03-18 21:49:24 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil> Duration:* 21.78745569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........................................XX.....XXX..X..XXX] duration: 30.003427132s trace: https://jaeger.dev.storj.io/trace/251f894af766eb83 FAIL (2026-03-18 21:49:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:48:32.645171629 (197h37m35.660841237s) [upload=189796]: Starting upload Mar 18 21:48:32.645326878 (197h37m35.660996481s) [upload=189796]: Done starting upload Mar 18 21:48:32.645418130 (197h37m35.661087732s) [upload=189796]: Uploading segments... Mar 18 21:48:32.645810289 (197h37m35.661479903s) [upload=189796]: Got next segment. Inline: false Mar 18 21:48:32.645974442 (197h37m35.661644261s) [upload=189796]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:48:32.645981851 (197h37m35.661651455s) [upload=189796]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:48:45.066247463 (197h37m48.081917068s) [seg_pos={0 0}, upload=189796]: Begin upload segment... Mar 18 21:48:45.066313671 (197h37m48.081983272s) [seg_pos={0 0}, upload=189796]: Done begin upload segment. Mar 18 21:48:45.066379707 (197h37m48.082049312s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.066410147 (197h37m48.08207974s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.066971840 (197h37m48.082641444s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.067083170 (197h37m48.082752778s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.067554140 (197h37m48.083223754s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.068499259 (197h37m48.084168865s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.068748780 (197h37m48.084418385s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.069466321 (197h37m48.085135933s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.069654213 (197h37m48.085323824s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.070484066 (197h37m48.086153671s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.071245408 (197h37m48.086915016s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.071594691 (197h37m48.087264302s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.072277538 (197h37m48.087947139s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074170774 (197h37m48.08984036s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074261023 (197h37m48.089930611s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.074812150 (197h37m48.090481742s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Uploading piece... Mar 18 21:48:45.668879318 (197h37m48.684548905s) [upload=189796]: Next returned nil segment Mar 18 21:48:45.668896432 (197h37m48.684566039s) [upload=189796]: Waiting for error group managing segments... Mar 18 21:48:45.797763739 (197h37m48.81343336s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.805326384 (197h37m48.820995984s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.806217578 (197h37m48.821887189s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.809363726 (197h37m48.825033341s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.811971344 (197h37m48.827640944s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.812960498 (197h37m48.828630103s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.813854402 (197h37m48.829524002s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.814801800 (197h37m48.830471401s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.823306084 (197h37m48.838975702s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.823905178 (197h37m48.839574778s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.824610241 (197h37m48.840279827s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.824966367 (197h37m48.840635966s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.831671478 (197h37m48.847341091s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil> Mar 18 21:48:45.831697992 (197h37m48.847367609s) [seg_pos={0 0}, upload=189796]: Satisfied cohort requirements after 13 pieces Mar 18 21:48:45.831975091 (197h37m48.8476447s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pvLo4F5ieDtepopg8Wi8jw6PKNeDCDvtH5iVNQXBUQwSXYxuD): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.831986640 (197h37m48.847656251s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837502196 (197h37m48.85317181s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.837522368 (197h37m48.853191978s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837711338 (197h37m48.853380951s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Mar 18 21:48:45.837720674 (197h37m48.853390286s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold Mar 18 21:48:45.837745572 (197h37m48.853415159s) [seg_pos={0 0}, upload=189796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:48:45.837833743 (197h37m48.853503354s) [upload=189796]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:48:45.837839430 (197h37m48.853509038s) [upload=189796]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:54.383402636 (197h37m57.399072251s) [upload=189796]: Done uploading segments... Mar 18 21:48:54.383428351 (197h37m57.399097962s) [upload=189796]: Upload finished. err: <nil> Duration: 30.003455447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............................................X....X.XXXX.XXX] duration: 30.005707553s trace: https://jaeger.dev.storj.io/trace/11dc3d42043d3b39 FAIL (2026-03-18 21:49:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00573699s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 35.583060045s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:48:44 UTC) orbiter warning: context deadline exceeded Duration: 35.583081095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [...........................................XX....XXXXXX..XX.] duration: 25.638183989s trace: https://jaeger.dev.storj.io/trace/6dbd9885f9de107d FAIL (2026-03-18 21:48:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 25.638202752s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.002030887s trace: https://jaeger.dev.storj.io/trace/593ce046ea9a8247 FAIL (2026-03-18 21:48:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002066647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [..................................................XXX..XX.XX] duration: 30.002564875s trace: https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5 FAIL (2026-03-18 21:48:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil> Duration:* 30.002590336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [............................................XX....XXXXXX..XX] duration: 30.001857974s trace: https://jaeger.dev.storj.io/trace/6dbd9885f9de107d FAIL (2026-03-18 21:48:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001872098s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................X....X.XXXX.XX] duration: 30.116176448s trace: https://jaeger.dev.storj.io/trace/2bc345e63b93f9dd FAIL (2026-03-18 21:48:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.116205418s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 51.36761058s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:48:09 UTC) orbiter warning: context deadline exceeded Duration: 51.367641032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................XX.....XXX..X..XX] duration: 30.003642955s trace: https://jaeger.dev.storj.io/trace/44d224acaa004e0 FAIL (2026-03-18 21:47:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:18.442808833 (197h36m21.458478442s) [upload=189779]: Starting upload Mar 18 21:47:18.442901121 (197h36m21.458570723s) [upload=189779]: Done starting upload Mar 18 21:47:18.442976117 (197h36m21.458645717s) [upload=189779]: Uploading segments... Mar 18 21:47:18.443606581 (197h36m21.459276181s) [upload=189779]: Got next segment. Inline: false Mar 18 21:47:18.443680474 (197h36m21.459350077s) [upload=189779]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:47:18.443699000 (197h36m21.459368603s) [upload=189779]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:47:26.699422791 (197h36m29.715092406s) [seg_pos={0 0}, upload=189779]: Begin upload segment... Mar 18 21:47:26.699491248 (197h36m29.715160841s) [seg_pos={0 0}, upload=189779]: Done begin upload segment. Mar 18 21:47:26.699526331 (197h36m29.715195943s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702077766 (197h36m29.717747375s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702861330 (197h36m29.71853093s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.702891630 (197h36m29.718561232s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703323314 (197h36m29.718992924s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703352656 (197h36m29.71902226s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703987436 (197h36m29.719657047s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.703997584 (197h36m29.719667195s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.704537437 (197h36m29.720207043s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.704977460 (197h36m29.720647062s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.706372766 (197h36m29.722042367s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.707008899 (197h36m29.722678497s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.707089684 (197h36m29.722759294s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.708051043 (197h36m29.723720636s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.709186798 (197h36m29.724856393s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:26.712310630 (197h36m29.727980233s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Uploading piece... Mar 18 21:47:27.306615628 (197h36m30.322285255s) [upload=189779]: Next returned nil segment Mar 18 21:47:27.306668889 (197h36m30.322338504s) [upload=189779]: Waiting for error group managing segments... Mar 18 21:47:27.442419844 (197h36m30.458089445s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.442434040 (197h36m30.458103645s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.444434343 (197h36m30.460103947s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.445531395 (197h36m30.46120101s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.445728504 (197h36m30.461398114s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.448910049 (197h36m30.46457967s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.451992678 (197h36m30.467662287s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.452128814 (197h36m30.467798432s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.454127332 (197h36m30.469796945s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.455872225 (197h36m30.471541839s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.458476641 (197h36m30.474146241s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.461198518 (197h36m30.47686815s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.466630469 (197h36m30.482300072s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil> Mar 18 21:47:27.466653132 (197h36m30.482322737s) [seg_pos={0 0}, upload=189779]: Satisfied cohort requirements after 13 pieces Mar 18 21:47:27.466926351 (197h36m30.482595955s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.466933928 (197h36m30.482603527s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.466972928 (197h36m30.482642554s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.466988039 (197h36m30.482657651s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.467169807 (197h36m30.482839418s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF Mar 18 21:47:27.467179258 (197h36m30.482848868s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold Mar 18 21:47:27.467211812 (197h36m30.482881442s) [seg_pos={0 0}, upload=189779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:47:27.467312886 (197h36m30.482982494s) [upload=189779]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:47:27.467320096 (197h36m30.482989711s) [upload=189779]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:47:35.897519767 (197h36m38.913189375s) [upload=189779]: Done uploading segments... Mar 18 21:47:35.897549016 (197h36m38.91321862s) [upload=189779]: Upload finished. err: <nil> Duration: 30.003669693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.000666722s trace: https://jaeger.dev.storj.io/trace/549c36531950e590 FAIL (2026-03-18 21:47:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000823006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................................X....X.XXXX.X] duration: 30.004707963s trace: https://jaeger.dev.storj.io/trace/1799e0deebe9782a FAIL (2026-03-18 21:47:27 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004736265s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XX....XXXXXX..X] duration: 30.001842368s trace: https://jaeger.dev.storj.io/trace/f51217a9a46a2f1 FAIL (2026-03-18 21:47:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001856253s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XXX..XX.X] duration: 30.001896226s trace: https://jaeger.dev.storj.io/trace/79f59160b77ec97a FAIL (2026-03-18 21:47:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:44.213562861 (197h35m47.229232461s) [upload=189770]: Starting upload Mar 18 21:46:44.213634624 (197h35m47.229304213s) [upload=189770]: Done starting upload Mar 18 21:46:44.213688107 (197h35m47.229357718s) [upload=189770]: Uploading segments... Mar 18 21:46:44.213783933 (197h35m47.229453544s) [upload=189770]: Got next segment. Inline: true Mar 18 21:46:44.213793186 (197h35m47.229462797s) [upload=189770]: Waiting for error group managing segments... Mar 18 21:46:44.213873893 (197h35m47.229543493s) [upload=189770]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:44.213881750 (197h35m47.229551356s) [upload=189770]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:46:44.213887721 (197h35m47.229557323s) [upload=189770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:56.859337880 (197h35m59.875007483s) [upload=189770]: Done uploading segments... Mar 18 21:46:56.859357249 (197h35m59.875026849s) [upload=189770]: Upload finished. err: <nil> Duration:* 30.001921231s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.001462524s trace: https://jaeger.dev.storj.io/trace/6f36e79e78a01382 FAIL (2026-03-18 21:47:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001491533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................................XX.....XXX..X..X] duration: 30.002391262s trace: https://jaeger.dev.storj.io/trace/3deeefe5cb22d838 FAIL (2026-03-18 21:47:02 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:33.016860025 (197h35m36.032529635s) [upload=189768]: Starting upload Mar 18 21:46:33.016989456 (197h35m36.032659062s) [upload=189768]: Done starting upload Mar 18 21:46:33.017052165 (197h35m36.03272177s) [upload=189768]: Uploading segments... Mar 18 21:46:33.017475356 (197h35m36.033144957s) [upload=189768]: Got next segment. Inline: false Mar 18 21:46:33.017539668 (197h35m36.03320927s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:46:33.017546318 (197h35m36.03321592s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:46:37.290282605 (197h35m40.305952212s) [seg_pos={0 0}, upload=189768]: Begin upload segment... Mar 18 21:46:37.290387539 (197h35m40.30605714s) [seg_pos={0 0}, upload=189768]: Done begin upload segment. Mar 18 21:46:37.290453240 (197h35m40.306122851s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.290923120 (197h35m40.30659273s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.291701810 (197h35m40.307371663s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292246979 (197h35m40.30791658s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292332991 (197h35m40.308002591s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292727864 (197h35m40.308397475s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.292767367 (197h35m40.308436977s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293090621 (197h35m40.308760221s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293273873 (197h35m40.308943484s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293460336 (197h35m40.309129947s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293769369 (197h35m40.309438976s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.293841511 (197h35m40.309511116s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294279896 (197h35m40.309949487s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.294826369 (197h35m40.310495979s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295420929 (197h35m40.311090529s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.295975630 (197h35m40.311645237s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Uploading piece... Mar 18 21:46:37.895634454 (197h35m40.911304055s) [upload=189768]: Next returned nil segment Mar 18 21:46:37.895651552 (197h35m40.911321163s) [upload=189768]: Waiting for error group managing segments... Mar 18 21:46:38.034797744 (197h35m41.05046737s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.034962121 (197h35m41.050631723s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.035419184 (197h35m41.051088784s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036603154 (197h35m41.052272755s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.036707110 (197h35m41.052376712s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.037782694 (197h35m41.053452295s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039517974 (197h35m41.055187575s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.039929620 (197h35m41.055599222s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040264640 (197h35m41.055934243s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.040742968 (197h35m41.056412577s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.051123109 (197h35m41.066792713s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.056784301 (197h35m41.072453905s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082796373 (197h35m41.098465981s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil> Mar 18 21:46:38.082828521 (197h35m41.098498126s) [seg_pos={0 0}, upload=189768]: Satisfied cohort requirements after 13 pieces Mar 18 21:46:38.083138766 (197h35m41.098808369s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W4YnnqptDeH7t6QVMo988zAyAZ3Kj3dFE1yaxjttuWjrMvKeM): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083150948 (197h35m41.098820558s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083217334 (197h35m41.098886945s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083248682 (197h35m41.098918292s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083440350 (197h35m41.099109956s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz): protocol: expected piece hash; context canceled; EOF Mar 18 21:46:38.083448598 (197h35m41.099118197s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold Mar 18 21:46:38.083471664 (197h35m41.09914126s) [seg_pos={0 0}, upload=189768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:46:38.083578356 (197h35m41.099247957s) [upload=189768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:46:38.083588526 (197h35m41.099258126s) [upload=189768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:46:54.701340920 (197h35m57.717010532s) [upload=189768]: Done uploading segments... Mar 18 21:46:54.701366952 (197h35m57.717036561s) [upload=189768]: Upload finished. err: <nil> Duration: 30.002419338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..............................................XX....XXXXXX..] duration: 21.708301964s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 21.708319921s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 32.267539216s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:39 UTC) orbiter warning: context deadline exceeded Duration: 32.267572524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................X....X.XXXX.] duration: 26.620919711s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:46:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 26.621007176s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [....................................................XXX..XX.] duration: 21.726106356s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:46:15 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 21.726220195s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.002293216s trace: https://jaeger.dev.storj.io/trace/78d2ae33cae77162 FAIL (2026-03-18 21:46:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002342162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 47.828840268s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:46:06 UTC) orbiter warning: context deadline exceeded Duration: 47.828854707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [...............................................XX....XXXXXX.] duration: 26.034862831s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:46:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 26.034895855s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XX.....XXX..X..] duration: 22.959785305s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:46:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 22.959816925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................X....X.XXXX] duration: 30.004684283s trace: https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96 FAIL (2026-03-18 21:45:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00472724s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.00101487s trace: https://jaeger.dev.storj.io/trace/332ada5a7badd8c6 FAIL (2026-03-18 21:45:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001043189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.182926396s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:45:18 UTC) orbiter warning: context deadline exceeded Duration: 36.182954232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX.....XXX..X.] duration: 27.136646187s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 27.136676097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XXX..XX] duration: 30.00119625s trace: https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89 FAIL (2026-03-18 21:45:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil> Duration:* 30.001237538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................X....X.XXX] duration: 30.312073232s trace: https://jaeger.dev.storj.io/trace/318408f9135b4ebe FAIL (2026-03-18 21:44:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.312088323s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [................................................XX....XXXXXX] duration: 30.001095386s trace: https://jaeger.dev.storj.io/trace/1c6f491aba147b1 FAIL (2026-03-18 21:44:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001121943s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000879906s trace: https://jaeger.dev.storj.io/trace/29e3b11860b462ab FAIL (2026-03-18 21:44:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000920376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX.....XXX..X] duration: 30.003061155s trace: https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb FAIL (2026-03-18 21:44:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments... Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment... Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment. Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece... Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments... Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil> Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments... Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil> Duration: 30.003086874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................X....X.XX] duration: 30.004686512s trace: https://jaeger.dev.storj.io/trace/741f8a58ad0c146 FAIL (2026-03-18 21:44:29 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004715111s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000849654s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 21:44:28 UTC) orbiter warning: context deadline exceeded Duration: 1m0.00086881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX..X] duration: 30.001426519s trace: https://jaeger.dev.storj.io/trace/5aa167875471c0f5 FAIL (2026-03-18 21:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:43:50.615014606 (197h32m53.630684213s) [upload=189728]: Starting upload Mar 18 21:43:50.615152702 (197h32m53.630822309s) [upload=189728]: Done starting upload Mar 18 21:43:50.615302807 (197h32m53.630972394s) [upload=189728]: Uploading segments... Mar 18 21:43:50.615408894 (197h32m53.631078495s) [upload=189728]: Got next segment. Inline: true Mar 18 21:43:50.615417472 (197h32m53.631087079s) [upload=189728]: Waiting for error group managing segments... Mar 18 21:43:50.615502560 (197h32m53.63117217s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:43:50.615509793 (197h32m53.631179394s) [upload=189728]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:43:50.615514503 (197h32m53.631184104s) [upload=189728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:44:20.332317413 (197h33m23.347987024s) [upload=189728]: Done uploading segments... Mar 18 21:44:20.332388334 (197h33m23.348057949s) [upload=189728]: Upload finished. err: metaclient: context deadline exceeded Duration:* 30.001466595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................................XX.....XXX..] duration: 22.966917102s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:44:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 22.966938471s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXX] duration: 5.003411737s trace: https://jaeger.dev.storj.io/trace/7ff31ea21f5ca509 FAIL (2026-03-18 21:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003428863s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)