Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..................X....X.XXXX.XXX...XXXX....................] duration: 2.317883623s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:10:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.31791627s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...................XXX..XX.XX..XX...........................] duration: 978.273319ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:10:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 978.370539ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 2.133663343s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:09:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.133695785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...................X....X.XXXX.XXX...XXXX...................] duration: 2.092956454s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:09:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.092984595s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............XX.....XXX..X..XXX.XXXX.XX.....................] duration: 2.395176215s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:09:37 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.395202211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...............XX....XXXXXX..XX.XXXXXXX.....................] duration: 928.385238ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:09:28 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 928.41723ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 1.736064678s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:09:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.73609568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.476976282s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:09:17 UTC) orbiter warning: context deadline exceeded Duration: 3.47700406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [....................XXX..XX.XX..XX..........................] duration: 958.676225ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:09:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 958.707447ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....................X....X.XXXX.XXX...XXXX..................] duration: 1.973200803s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:08:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.973332052s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............XX.....XXX..X..XXX.XXXX.XX....................] duration: 2.370868259s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:08:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.370903121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [................XX....XXXXXX..XX.XXXXXXX....................] duration: 947.607455ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:08:45 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 947.637628ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.573884324s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:08:34 UTC) orbiter warning: context deadline exceeded Duration: 3.573913063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 1.639703598s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:08:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.639735272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.....................XXX..XX.XX..XX.........................] duration: 952.221798ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:08:12 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 952.253594ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....................X....X.XXXX.XXX...XXXX.................] duration: 2.169426291s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:08:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.169453494s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.................XX....XXXXXX..XX.XXXXXXX...................] duration: 950.715677ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:08:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 950.745015ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...............XX.....XXX..X..XXX.XXXX.XX...................] duration: 2.356707278s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:07:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.356809219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [......................XXX..XX.XX..XX........................] duration: 937.980582ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:07:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 938.01749ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.863742882s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:07:41 UTC) orbiter warning: context deadline exceeded Duration: 2.863781746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.......................XXX..XX.XX..XX.......................] duration: 1.050834707s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:07:28 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.050867418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 2.114624588s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.114652957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......................X....X.XXXX.XXX...XXXX................] duration: 1.994909919s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:07:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.994943283s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.800861283s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:07:01 UTC) orbiter warning: context deadline exceeded Duration: 2.801273816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX..................] duration: 1.685160146s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:07:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685186303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................XX.....XXX..X..XXX.XXXX.XX..................] duration: 2.365408065s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:06:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.365435675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [........................XXX..XX.XX..XX......................] duration: 955.538834ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:06:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 955.563884ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [..................XX....XXXXXX..XX.XXXXXXX..................] duration: 940.330189ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:06:52 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 940.362261ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.........................XXX..XX.XX..XX.....................] duration: 955.236841ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:06:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 955.265263ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......................X....X.XXXX.XXX...XXXX...............] duration: 2.256750673s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:06:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.25677687s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.868672418s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:06:20 UTC) orbiter warning: context deadline exceeded Duration: 2.868698009s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [...................XX....XXXXXX..XX.XXXXXXX.................] duration: 960.340528ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:06:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 960.381708ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX.................] duration: 1.898128474s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:06:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.898160222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX.....XXX..X..XXX.XXXX.XX.................] duration: 2.460088689s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:06:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.460111704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........................X....X.XXXX.XXX...XXXX..............] duration: 2.271022546s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:05:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.27105308s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..........................XXX..XX.XX..XX....................] duration: 1.031347805s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:05:58 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.03136657s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX................] duration: 1.837035829s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.837070465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................XX.....XXX..X..XXX.XXXX.XX................] duration: 2.348831101s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:05:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.348930186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [....................XX....XXXXXX..XX.XXXXXXX................] duration: 920.722924ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:05:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 920.752872ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.049671147s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:05:30 UTC) orbiter warning: context deadline exceeded Duration: 3.049700035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...................XX.....XXX..X..XXX.XXXX.XX...............] duration: 2.359080613s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:05:23 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.359110991s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX...............] duration: 1.68230663s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:05:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.682352255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX..............] duration: 2.261975059s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:05:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.262004892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [.....................XX....XXXXXX..XX.XXXXXXX...............] duration: 949.043733ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:04:56 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 949.07971ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.809284496s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:04:52 UTC) orbiter warning: context deadline exceeded Duration: 2.809308387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...........................XXX..XX.XX..XX...................] duration: 1.041325877s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:04:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.041376563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................X....X.XXXX.XXX...XXXX.............] duration: 2.269057842s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:04:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.269086556s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 17 out of 60 is failed (threshold=3) [......................XX....XXXXXX..XX.XXXXXXX..............] duration: 925.810709ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:04:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 925.835559ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.452970278s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:04:37 UTC) orbiter warning: context deadline exceeded Duration: 3.453011932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 9 out of 60 is failed (threshold=3) [............................XXX..XX.XX..XX..................] duration: 976.377022ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:04:20 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 976.425445ms core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)