Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 16 out of 60 is failed (threshold=3) [X....XXXXXX..XX.XXXXXXX.....................................] duration: 922.896298ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:20:59 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 922.925698ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [......XXX..XX.XX..XX........................................] duration: 974.284479ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:20:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 974.323715ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX......................................] duration: 1.557996469s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.55805422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.858626268s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:20:51 UTC) orbiter warning: context deadline exceeded Duration: 2.858683175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.....................................] duration: 1.815033224s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:20:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.815067244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....XXX..X..XXX.XXXX.XX....................................] duration: 2.522536309s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:20:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.522624693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X....X.XXXX.XXX...XXXX...................................] duration: 2.216850249s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:20:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.217131231s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X....X.XXXX.XXX...XXXX..................................] duration: 2.215515188s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:20:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.215548481s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....XXX..X..XXX.XXXX.XX...................................] duration: 2.519148789s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:20:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.51919433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [XX....XXXXXX..XX.XXXXXXX....................................] duration: 956.120894ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:20:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 956.154323ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.......XXX..XX.XX..XX.......................................] duration: 947.350019ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:19:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 947.378852ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.384741648s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:19:43 UTC) orbiter warning: context deadline exceeded Duration: 3.385042486s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X....X.XXXX.XXX...XXXX.................................] duration: 2.212067531s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:19:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.212095929s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.....XXX..X..XXX.XXXX.XX..................................] duration: 2.370625684s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:19:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.370665712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX....................................] duration: 1.708837666s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:19:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.708889396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.XX....XXXXXX..XX.XXXXXXX...................................] duration: 987.376909ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:19:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 987.405612ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXX...................................] duration: 1.685743413s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:19:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685775745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX.....XXX..X..XXX.XXXX.XX.................................] duration: 2.447179405s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:18:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.447208285s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [........XXX..XX.XX..XX......................................] duration: 966.434804ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:18:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 966.463629ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXX..................................] duration: 1.624853947s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:18:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.624894291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X....X.XXXX.XXX...XXXX................................] duration: 2.228200212s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:18:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.228227277s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.743439809s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:18:32 UTC) orbiter warning: context deadline exceeded Duration: 2.743540331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [..XX....XXXXXX..XX.XXXXXXX..................................] duration: 911.337801ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:18:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 911.385747ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.........XXX..XX.XX..XX.....................................] duration: 965.602679ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:18:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 965.635037ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X....X.XXXX.XXX...XXXX...............................] duration: 2.196109256s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:18:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.196138993s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX.....XXX..X..XXX.XXXX.XX................................] duration: 2.475199483s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:18:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.475229891s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.474000968s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:18:04 UTC) orbiter warning: context deadline exceeded Duration: 3.474034243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..........XXX..XX.XX..XX....................................] duration: 980.001984ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:17:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 980.034695ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.739461194s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:17:42 UTC) orbiter warning: context deadline exceeded Duration: 2.739491506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...XX....XXXXXX..XX.XXXXXXX.................................] duration: 941.586195ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:17:30 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 941.624911ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X....X.XXXX.XXX...XXXX..............................] duration: 2.498591423s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:17:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.498622704s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXX.................................] duration: 1.65165443s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.652207103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...XX.....XXX..X..XXX.XXXX.XX...............................] duration: 2.359683584s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:17:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.359727579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [....XX....XXXXXX..XX.XXXXXXX................................] duration: 902.794072ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:17:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 902.824963ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.729287379s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:17:09 UTC) orbiter warning: context deadline exceeded Duration: 2.729316518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X....X.XXXX.XXX...XXXX.............................] duration: 1.952138693s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:16:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.952172914s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........X....X.XXXX.XXX...XXXX............................] duration: 2.134710545s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:16:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.134740404s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXXXX................................] duration: 1.7724919s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:16:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.772519124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX.....XXX..X..XXX.XXXX.XX..............................] duration: 2.339674207s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:16:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.339716702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...........XXX..XX.XX..XX...................................] duration: 957.433733ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:16:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 957.452816ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.....XX....XXXXXX..XX.XXXXXXX...............................] duration: 901.893059ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:16:24 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 901.932246ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........X....X.XXXX.XXX...XXXX...........................] duration: 2.200685442s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:16:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.200718793s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.763962904s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:16:05 UTC) orbiter warning: context deadline exceeded Duration: 2.763992088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [......XX....XXXXXX..XX.XXXXXXX..............................] duration: 932.406948ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:15:55 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 932.433761ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX.....XXX..X..XXX.XXXX.XX.............................] duration: 2.340790854s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:15:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.340819705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [............XXX..XX.XX..XX..................................] duration: 928.89668ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:15:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 928.923045ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 1.677897789s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:15:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.677917495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XX.....XXX..X..XXX.XXXX.XX............................] duration: 2.349041973s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:15:22 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.34907141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.265992326s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:15:16 UTC) orbiter warning: context deadline exceeded Duration: 3.266018965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [............X....X.XXXX.XXX...XXXX..........................] duration: 2.387076118s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:15:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.387106463s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)