Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.858830356s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:27:02 UTC) orbiter warning: context deadline exceeded Duration: 2.858865424s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX...............................................] duration: 2.124721449s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.124748963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XXX.XXXX.XX...............................................] duration: 2.4704152s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:26:59 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.470444938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 13 out of 60 is failed (threshold=3) [XXXX..XX.XXXXXXX............................................] duration: 953.236456ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:26:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 953.265145ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXXX.XXX...XXXX............................................] duration: 2.196110455s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:26:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.196139192s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX..............................................] duration: 2.734437071s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:26:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.734467368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..XXX.XXXX.XX..............................................] duration: 2.354728154s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:26:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.354831776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [XXX..XX.XX..XX..............................................] duration: 977.637739ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:26:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 977.667008ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.78955963s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:26:05 UTC) orbiter warning: context deadline exceeded Duration: 2.789593799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..XXX.XXXX.XX.............................................] duration: 2.340315531s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:25:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.340343916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 14 out of 60 is failed (threshold=3) [XXXXX..XX.XXXXXXX...........................................] duration: 968.958961ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:25:53 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 969.00085ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.XXXX.XXX...XXXX...........................................] duration: 2.249290783s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:25:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.249313989s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..XXX.XXXX.XX............................................] duration: 2.446722307s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:25:36 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.446754286s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.960494226s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:25:25 UTC) orbiter warning: context deadline exceeded Duration: 2.960519455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.............................................] duration: 1.644154834s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:25:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.644181099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXX..XX.XXXXXXX..........................................] duration: 972.24546ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:25:19 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 972.27431ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.XXXX.XXX...XXXX..........................................] duration: 2.397108759s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:25:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.397135073s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.XXX..XX.XX..XX.............................................] duration: 1.157432939s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:25:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.157529794s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.905776005s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:25:02 UTC) orbiter warning: context deadline exceeded Duration: 2.905808085s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.XXXX.XXX...XXXX.........................................] duration: 2.284234824s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:24:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.284261618s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX............................................] duration: 1.618885359s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:24:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.618916051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X..XXX.XXXX.XX...........................................] duration: 2.382356164s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:24:46 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.382395857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [.XXXXXX..XX.XXXXXXX.........................................] duration: 973.428908ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:24:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 973.464542ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..XXX..XX.XX..XX............................................] duration: 1.023304363s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:24:10 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.023339694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX...........................................] duration: 1.712400777s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:24:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.712440127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.94764759s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:24:08 UTC) orbiter warning: context deadline exceeded Duration: 2.947679487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.XXXX.XXX...XXXX........................................] duration: 2.233773629s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:24:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.233808931s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX..........................................] duration: 1.586469237s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:23:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.586499159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..X..XXX.XXXX.XX..........................................] duration: 2.420105631s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:23:31 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.420136131s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.........................................] duration: 2.278710839s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:23:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.278751776s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [..XXXXXX..XX.XXXXXXX........................................] duration: 953.742219ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:23:17 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 953.843334ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX..X..XXX.XXXX.XX.........................................] duration: 2.459892955s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:23:11 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.459923527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...XXX..XX.XX..XX...........................................] duration: 910.353111ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:22:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 910.3815ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.188670752s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:22:55 UTC) orbiter warning: context deadline exceeded Duration: 3.188699371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.XXXX.XXX...XXXX.......................................] duration: 2.266483998s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:22:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.266512646s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [...XXXXXX..XX.XXXXXXX.......................................] duration: 903.737838ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:22:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 903.771196ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [....XXX..XX.XX..XX..........................................] duration: 962.046031ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:22:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 962.101236ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX........................................] duration: 2.148670374s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:22:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.148759755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX..X..XXX.XXXX.XX........................................] duration: 2.452700023s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:22:34 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.452734597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....X.XXXX.XXX...XXXX......................................] duration: 2.464695843s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:22:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.464746836s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.909248735s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:22:29 UTC) orbiter warning: context deadline exceeded Duration: 2.909287879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX..X..XXX.XXXX.XX.......................................] duration: 2.395718554s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:22:16 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.395756204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.....XXX..XX.XX..XX.........................................] duration: 982.972702ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:22:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 983.005814ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.423476038s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:21:54 UTC) orbiter warning: context deadline exceeded Duration: 3.423507719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....X.XXXX.XXX...XXXX.....................................] duration: 2.16304214s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:21:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.163092357s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.......................................] duration: 1.585874879s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:21:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.586038475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XXX..X..XXX.XXXX.XX......................................] duration: 2.397908403s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:21:41 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.398008917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 15 out of 60 is failed (threshold=3) [....XXXXXX..XX.XXXXXXX......................................] duration: 909.267419ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:21:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 909.295794ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XXX..X..XXX.XXXX.XX.....................................] duration: 2.429450534s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:21:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.42948933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X....X.XXXX.XXX...XXXX....................................] duration: 2.426702172s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:21:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.426736926s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)