Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 2.111839751s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:34:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.111880926s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.XXXXXXX....................................................] duration: 1.01137126s trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:34:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.011402588s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 921.647076ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:34:17 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 921.672051ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 2.219498366s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:34:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.219532705s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.XX........................................................] duration: 2.337057822s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:33:57 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.337088449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 2.208723276s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:33:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.208748996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 2.090062422s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:33:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.090103809s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [X..XX.......................................................] duration: 1.012205225s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:33:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.012257664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 8 out of 60 is failed (threshold=3) [X.XXXXXXX...................................................] duration: 940.261949ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:33:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 940.293096ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...XXXX....................................................] duration: 2.175467851s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:33:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.175551983s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.XX.......................................................] duration: 2.348759566s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:32:51 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.348796467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XXXXX.......................................................] duration: 1.745564671s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:32:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.745606768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [XX..XX......................................................] duration: 962.794951ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:32:35 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 962.904811ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX...XXXX...................................................] duration: 1.94134683s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:32:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.941376745s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XXX.XX......................................................] duration: 2.426379245s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:32:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.426410354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [XX.XXXXXXX..................................................] duration: 914.67421ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:31:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 914.714123ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 4 out of 60 is failed (threshold=3) [.XX..XX.....................................................] duration: 1.000194079s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:31:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.000248101s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XXXXXX......................................................] duration: 1.66467325s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.664711578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXXX.XX.....................................................] duration: 2.404361603s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:31:38 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.404395575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.XX.XXXXXXX.................................................] duration: 934.015151ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:31:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 934.040978ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXX...XXXX..................................................] duration: 2.085284053s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:31:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.085313528s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XXXXXXX.....................................................] duration: 2.1697205s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:31:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.169750961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 9 out of 60 is failed (threshold=3) [..XX.XXXXXXX................................................] duration: 985.319932ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:31:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 985.355418ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XXXX.XX....................................................] duration: 2.389019652s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:31:05 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.38904528s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXXXXXX....................................................] duration: 1.761430588s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:30:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.76146487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 5 out of 60 is failed (threshold=3) [X.XX..XX....................................................] duration: 965.738703ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:30:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 965.796946ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XXX...XXXX.................................................] duration: 2.582538212s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:30:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.582560772s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXXXXXXXX...................................................] duration: 1.722801348s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:30:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.722827562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 10 out of 60 is failed (threshold=3) [X..XX.XXXXXXX...............................................] duration: 1.022776172s trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:30:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.022818576s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.XXXX.XX...................................................] duration: 2.40747742s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:30:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.407506954s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XXXXXXXXXX..................................................] duration: 1.750275002s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.750303309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [XX.XX..XX...................................................] duration: 1.024323967s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:29:48 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.024342407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX.XXXX.XX..................................................] duration: 2.390246538s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:29:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.390276181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.XXX...XXXX................................................] duration: 2.334067016s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:29:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.334095021s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [.XX.XX..XX..................................................] duration: 913.524468ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:29:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 913.564501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.XXX...XXXX...............................................] duration: 2.054251574s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:29:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.054282941s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 11 out of 60 is failed (threshold=3) [XX..XX.XXXXXXX..............................................] duration: 989.105771ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:29:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 989.137938ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.882058092s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:28:52 UTC) orbiter warning: context deadline exceeded Duration: 2.882082811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXX.XXXX.XX.................................................] duration: 2.411650647s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:28:47 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.41167778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..XX.XX..XX.................................................] duration: 1.055990269s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:28:42 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.056017802s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XXXXXXXXXXX.................................................] duration: 1.748440848s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:28:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.74863187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.833881857s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:28:35 UTC) orbiter warning: context deadline exceeded Duration: 2.834143461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX.XXX...XXXX..............................................] duration: 2.338136932s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:28:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.338170458s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 7 out of 60 is failed (threshold=3) [X..XX.XX..XX................................................] duration: 1.02271768s trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:28:21 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 1.022747263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XXX.XXXX.XX................................................] duration: 2.427856955s trace: https://jaeger.dev.storj.io/trace/75e0f0868fd0d72 FAIL (2026-03-18 22:28:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:54:18.149554843 (197h43m21.165224435s) [upload=189883]: Starting upload Mar 18 21:54:18.149629705 (197h43m21.165299296s) [upload=189883]: Done starting upload Mar 18 21:54:18.149726090 (197h43m21.16539568s) [upload=189883]: Uploading segments... Mar 18 21:54:18.149968905 (197h43m21.165638498s) [upload=189883]: Got next segment. Inline: false Mar 18 21:54:18.150024167 (197h43m21.165693769s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:54:18.150031397 (197h43m21.165700997s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:54:26.308528312 (197h43m29.324197912s) [seg_pos={0 0}, upload=189883]: Begin upload segment... Mar 18 21:54:26.308650989 (197h43m29.324320588s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.309947151 (197h43m29.325616759s) [seg_pos={0 0}, upload=189883]: Done begin upload segment. Mar 18 21:54:26.310024877 (197h43m29.325694487s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.310648388 (197h43m29.326318s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.311260449 (197h43m29.32693006s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.312476969 (197h43m29.328146592s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.313742039 (197h43m29.329411642s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.314129731 (197h43m29.32979934s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.315306083 (197h43m29.330975691s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.316537735 (197h43m29.332207343s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.317296012 (197h43m29.332965624s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.318422029 (197h43m29.334091636s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.319037770 (197h43m29.334707382s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.320223299 (197h43m29.335892922s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321492900 (197h43m29.337162507s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.321765536 (197h43m29.337435143s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.323594932 (197h43m29.339264544s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Uploading piece... Mar 18 21:54:26.935519647 (197h43m29.951189247s) [upload=189883]: Next returned nil segment Mar 18 21:54:26.935535154 (197h43m29.951204754s) [upload=189883]: Waiting for error group managing segments... Mar 18 21:54:27.064763826 (197h43m30.080433427s) [noise=true , address=77.90.31.19:4842 , piece_id=PWAEYFLRCTKDT4H7AG7LAUJKUWWU45SOQTPC4PHUIFUSFDRLBR7A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066336450 (197h43m30.08200605s) [noise=true , address=45.91.116.8:4080 , piece_id=CN3YXD7WEAUY57YFADWOVHPPANYZJBRT4VPJ6INVCISIFKB7CKCQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.066710529 (197h43m30.082380129s) [noise=true , address=77.90.31.10:4518 , piece_id=HRYWJZEFJJ5NV6DFG7XM6VY2AFWNF5D6CFTPESBTI42J5WAW7MXA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067163041 (197h43m30.082832641s) [noise=true , address=145.79.227.14:4841 , piece_id=GZXEN7OM7IAP5PSXSVR5SVSDKHD33ISGVVIGWQQPFS4HGLJMUROQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067539264 (197h43m30.083208864s) [noise=true , address=45.91.116.14:4193 , piece_id=KWARE3PWKRLL3DCJ6FX47725P57CGFFS4DR6UOTHDJZQQOKJSKHQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.067890346 (197h43m30.08355995s) [noise=true , address=45.91.116.11:4118 , piece_id=YSU73JQIBOQBE5YQ7LQKSQM6FURD5OLQRPPTS55WEEEVW4KMGLPQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.068825983 (197h43m30.084495583s) [noise=true , address=145.79.227.32:4833 , piece_id=I4MJO76NNIEOOP44VSTOOY2MDNSWR2JLDSGJWQ4TSF45SYEAF3KA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071631864 (197h43m30.087301453s) [noise=true , address=145.79.227.46:4606 , piece_id=IDZPEIF73YXZPU2U35NW2JBCIZXJ42APUZDZLBBH2QN6HP4YNRRA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.071767013 (197h43m30.087436614s) [noise=true , address=45.91.116.7:4009 , piece_id=DWR7XXGJVAAISBACLPLWVTKZXU3EJY6SXD7CNAW7XSL6O76AMLJA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.072116600 (197h43m30.0877862s) [noise=true , address=145.79.227.44:4195 , piece_id=K3YARWO7WM6N3QEC4ZJWJIGP6AWRSWGZQSBYH6UU3NP6M6QFHX3Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.073662705 (197h43m30.089332313s) [noise=true , address=77.90.31.13:4626 , piece_id=CLILSPUBHXP3VEGE6OQNFO7QZI564POQHZNN3267TIJVODMG4N2Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.081111494 (197h43m30.096781094s) [noise=true , address=45.91.116.4:4955 , piece_id=ZM2JV5BN5U3LKLPEQOOH4D42QHVZJFYPZCHFXU2SHXFSOJFXJVTA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085500985 (197h43m30.101170602s) [noise=true , address=145.79.227.45:4230 , piece_id=O2X5ADE7BIIR3HHLF2PETI6ZWYPQSU4SNV3BVBCVNT5CZ722SEDQ, seg_pos={0 0}, upload=189883]: Done uploading piece. err: <nil> Mar 18 21:54:27.085522039 (197h43m30.10119164s) [seg_pos={0 0}, upload=189883]: Satisfied cohort requirements after 13 pieces Mar 18 21:54:27.086116447 (197h43m30.101786058s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086202338 (197h43m30.101871943s) [noise=true , address=145.79.227.22:4515 , piece_id=GYGZ7KRMHQDTPI6XZRGMPOEYXRGXMY2FURVUUYUB7AA6CFEH326Q, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086610013 (197h43m30.102279622s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZGynTzsrzGcGzMCqbeMJkxWwhYtPc2piotu6dhkJzrr4FnXKd): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086645950 (197h43m30.10231555s) [noise=true , address=77.90.31.16:4738 , piece_id=E7IKMPRSNQXCACUY5HBRDPN4YZ37APUXQQZVHED4YQJGC7YPHZ4A, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.086923172 (197h43m30.102592787s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt): protocol: expected piece hash; context canceled; EOF Mar 18 21:54:27.086949914 (197h43m30.102619517s) [noise=true , address=145.79.227.43:4162 , piece_id=HIMVFURGSWUMODFHILTPC7DM3CLYZ4NSTOWLFIKN7B257EYLZTIA, seg_pos={0 0}, upload=189883]: Long tail cancellation due to optimal threshold Mar 18 21:54:27.087062900 (197h43m30.10273249s) [seg_pos={0 0}, upload=189883]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:54:27.087234507 (197h43m30.102904107s) [upload=189883]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:54:27.087240800 (197h43m30.1029104s) [upload=189883]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:54:39.677837110 (197h43m42.693506704s) [upload=189883]: Done uploading segments... Mar 18 21:54:39.677859307 (197h43m42.693528908s) [upload=189883]: Upload finished. err: <nil> Duration: 2.42789683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXXXXXXXXXXX................................................] duration: 2.171751235s trace: https://jaeger.dev.storj.io/trace/2c02709149f42d48 FAIL (2026-03-18 22:27:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.171780866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 12 out of 60 is failed (threshold=3) [XXX..XX.XXXXXXX.............................................] duration: 974.147618ms trace: https://jaeger.dev.storj.io/trace/1e14c800e54addc9 FAIL (2026-03-18 22:27:52 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 974.182063ms core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX.XXX...XXXX.............................................] duration: 2.042672323s trace: https://jaeger.dev.storj.io/trace/61dc9d7af1d4ec01 FAIL (2026-03-18 22:27:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.042701105s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 8 out of 60 is failed (threshold=3) [XX..XX.XX..XX...............................................] duration: 952.728452ms trace: https://jaeger.dev.storj.io/trace/65f276fda07fe2e FAIL (2026-03-18 22:27:33 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil> Duration:* 952.755833ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.897404769s trace: https://jaeger.dev.storj.io/trace/ebcefeaecc8948f FAIL (2026-03-18 22:27:33 UTC) orbiter warning: context deadline exceeded Duration: 2.897521534s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)