Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
14 out of 60 is failed (threshold=3) [.......................................XX....XXXXXX..XX.XXXX] duration:
30.001158041s trace:
https://jaeger.dev.storj.io/trace/597b0f7c6dbdd8ec
FAIL (2026-03-18 21:51:57 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.001309578s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration:
5.000747713s trace:
https://jaeger.dev.storj.io/trace/41ac61cf3e61df6
FAIL (2026-03-18 21:51:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000811405s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
39.534806715s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:51:27 UTC)
orbiter warning: context deadline exceeded
Duration: 39.534832604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
13 out of 60 is failed (threshold=3) [........................................XX....XXXXXX..XX.XXX] duration:
30.001339885s trace:
https://jaeger.dev.storj.io/trace/4bcb1866c69971b0
FAIL (2026-03-18 21:51:26 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.001356569s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
9 out of 60 is failed (threshold=3) [..............................................XXX..XX.XX..XX] duration:
30.005517978s trace:
https://jaeger.dev.storj.io/trace/65f276fda07fe2e
FAIL (2026-03-18 21:51:25 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:55.785119820 (197h39m58.800789429s) [upload=189833]: Starting upload Mar 18 21:50:55.785202430 (197h39m58.80087204s) [upload=189833]: Done starting upload Mar 18 21:50:55.785303739 (197h39m58.800973328s) [upload=189833]: Uploading segments... Mar 18 21:50:55.785387166 (197h39m58.801056775s) [upload=189833]: Got next segment. Inline: true Mar 18 21:50:55.785393596 (197h39m58.801063197s) [upload=189833]: Waiting for error group managing segments... Mar 18 21:50:55.785466465 (197h39m58.801136064s) [upload=189833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:50:55.785472300 (197h39m58.801141901s) [upload=189833]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:55.785497156 (197h39m58.801166757s) [upload=189833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:51:20.588512781 (197h40m23.604182387s) [upload=189833]: Done uploading segments... Mar 18 21:51:20.588537265 (197h40m23.604206873s) [upload=189833]: Upload finished. err: <nil>
Duration: 30.0055317s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................................X....X.XXXX.XXX...] duration:
22.665886003s trace:
https://jaeger.dev.storj.io/trace/11dc3d42043d3b39
FAIL (2026-03-18 21:51:12 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 22.665922301s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.......................................XX.....XXX..X..XXX.XX] duration:
30.003210823s trace:
https://jaeger.dev.storj.io/trace/6866a42fd61f5aeb
FAIL (2026-03-18 21:51:06 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:50:36.692726361 (197h39m39.708395969s) [upload=189828]: Starting upload
Mar 18 21:50:36.692813955 (197h39m39.708483565s) [upload=189828]: Done starting upload
Mar 18 21:50:36.692912555 (197h39m39.708582156s) [upload=189828]: Uploading segments...
Mar 18 21:50:36.693312194 (197h39m39.708981801s) [upload=189828]: Got next segment. Inline: false
Mar 18 21:50:36.693392117 (197h39m39.709061726s) [upload=189828]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:50:36.693415391 (197h39m39.709085001s) [upload=189828]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:50:40.907074880 (197h39m43.922744511s) [seg_pos={0 0}, upload=189828]: Begin upload segment...
Mar 18 21:50:40.907157303 (197h39m43.922826912s) [seg_pos={0 0}, upload=189828]: Done begin upload segment.
Mar 18 21:50:40.907204563 (197h39m43.922874163s) [noise=true , address=145.79.227.34:4906 , piece_id=5MA4X22RVBZ4QIYAVOYMZM6G3ZGKYXPHV4RKPGTWMKLYF4MIEEOQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.907222767 (197h39m43.922892368s) [noise=true , address=145.79.227.14:4841 , piece_id=CEEFSB3URJY4BRZBWHZ24HZDQDVAE65SU6FQXCAJ5YKCTXRFE34Q, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.907281856 (197h39m43.922951456s) [noise=true , address=145.79.227.15:4878 , piece_id=L5AAQGF4IBAXOQWDYQU2QI2ME5K2MIJMILWMI7C7SPISOSWTIDMA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.907968147 (197h39m43.92363775s) [noise=true , address=145.79.227.2:4119 , piece_id=V5MXFTXI3KKNQB5JHA4ZW6QAIC3TBHWSGHNG2ZIMCNJEX6QUSWPA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.908034573 (197h39m43.923704175s) [noise=true , address=145.79.227.25:4622 , piece_id=22IGX3OHI3EJ4CFWXSITKCIMUFZWOKABW4QJGADUY7GLGCB6FKPA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.908447413 (197h39m43.924117018s) [noise=true , address=145.79.227.17:4973 , piece_id=NGGRPASFGBSNH2XKU4L22H2TDKLMTRN2FGTCGOSWRHZ7SVMP725A, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.908801575 (197h39m43.924471183s) [noise=true , address=79.127.219.45:4854 , piece_id=IYHUNASM5KORNGBFMS4HOMBNJ2HKA42W3UKL4ZPX57PWYN3TPSYQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.910491118 (197h39m43.926160725s) [noise=true , address=145.79.227.33:4870 , piece_id=W2VK6RAQ6Y7FNG5SE7KWS4OWA2EYZJ54GQYKCFO3SNFVYMUG63YA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.910912291 (197h39m43.926581891s) [noise=true , address=145.79.227.24:4585 , piece_id=Q43NAMGYUSA5C44IMARW5IC6UTZHYRXAMNELVWKSBDFZB6XEQJ3A, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.911244195 (197h39m43.926913796s) [noise=true , address=79.127.219.41:4930 , piece_id=R7KNDLUIUAQUUTDGS3PU5CMTQUIFVVYIQU7CXVVIVGNJ57GQJTUA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.911575517 (197h39m43.927245118s) [noise=true , address=145.79.227.51:4412 , piece_id=6XOBSDHRJTSNZPPMSCIBYIKATVS76ZSJPEES5ZGJWRDDPVV5XMIQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.912177072 (197h39m43.927846676s) [noise=true , address=145.79.227.31:4798 , piece_id=S7DZDVOFQZKH52BV2WSXNYFKUXEKA47QFUUX3P33OUBIYHAMIRBA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.912589970 (197h39m43.928259571s) [noise=true , address=77.90.31.11:4556 , piece_id=W7RRHQHWD6NAPHCOUW2SF6254C3FHGNEGBATG43GOSRP2JUE3JOA, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.913003824 (197h39m43.928673425s) [noise=true , address=77.90.31.18:4808 , piece_id=R2OX6UA26PPFISSE22DFGV6CIZMDR7Z4QJ5UXHBPFLSWQD6SECOQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.913496720 (197h39m43.92916632s) [noise=true , address=145.79.227.46:4601 , piece_id=TPCX23W264SA26RCGG7X2KAMGNOOS2B6QSX65NTH25PKY3S2MZQQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:40.913999339 (197h39m43.929668944s) [noise=true , address=45.91.116.5:4300 , piece_id=4KYBWVZTBEGCRL4VCUY6OEJT6QEBO7GTKTU453T2A6MGLE734ICQ, seg_pos={0 0}, upload=189828]: Uploading piece...
Mar 18 21:50:41.517012158 (197h39m44.532681787s) [upload=189828]: Next returned nil segment
Mar 18 21:50:41.517038652 (197h39m44.532708289s) [upload=189828]: Waiting for error group managing segments...
Mar 18 21:50:41.651267591 (197h39m44.666937192s) [noise=true , address=79.127.219.45:4854 , piece_id=IYHUNASM5KORNGBFMS4HOMBNJ2HKA42W3UKL4ZPX57PWYN3TPSYQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.652009901 (197h39m44.667679509s) [noise=true , address=145.79.227.15:4878 , piece_id=L5AAQGF4IBAXOQWDYQU2QI2ME5K2MIJMILWMI7C7SPISOSWTIDMA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.657843360 (197h39m44.673512961s) [noise=true , address=145.79.227.14:4841 , piece_id=CEEFSB3URJY4BRZBWHZ24HZDQDVAE65SU6FQXCAJ5YKCTXRFE34Q, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.659302133 (197h39m44.67497174s) [noise=true , address=145.79.227.46:4601 , piece_id=TPCX23W264SA26RCGG7X2KAMGNOOS2B6QSX65NTH25PKY3S2MZQQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.660940099 (197h39m44.67660971s) [noise=true , address=145.79.227.17:4973 , piece_id=NGGRPASFGBSNH2XKU4L22H2TDKLMTRN2FGTCGOSWRHZ7SVMP725A, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.661730093 (197h39m44.6773997s) [noise=true , address=145.79.227.25:4622 , piece_id=22IGX3OHI3EJ4CFWXSITKCIMUFZWOKABW4QJGADUY7GLGCB6FKPA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.662533014 (197h39m44.678202618s) [noise=true , address=145.79.227.2:4119 , piece_id=V5MXFTXI3KKNQB5JHA4ZW6QAIC3TBHWSGHNG2ZIMCNJEX6QUSWPA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.662588045 (197h39m44.678257645s) [noise=true , address=145.79.227.51:4412 , piece_id=6XOBSDHRJTSNZPPMSCIBYIKATVS76ZSJPEES5ZGJWRDDPVV5XMIQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.663168676 (197h39m44.678838276s) [noise=true , address=145.79.227.24:4585 , piece_id=Q43NAMGYUSA5C44IMARW5IC6UTZHYRXAMNELVWKSBDFZB6XEQJ3A, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.664461167 (197h39m44.680130774s) [noise=true , address=45.91.116.5:4300 , piece_id=4KYBWVZTBEGCRL4VCUY6OEJT6QEBO7GTKTU453T2A6MGLE734ICQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.665796200 (197h39m44.681465801s) [noise=true , address=145.79.227.34:4906 , piece_id=5MA4X22RVBZ4QIYAVOYMZM6G3ZGKYXPHV4RKPGTWMKLYF4MIEEOQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.667095739 (197h39m44.682765351s) [noise=true , address=77.90.31.11:4556 , piece_id=W7RRHQHWD6NAPHCOUW2SF6254C3FHGNEGBATG43GOSRP2JUE3JOA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.669525775 (197h39m44.685195374s) [noise=true , address=145.79.227.33:4870 , piece_id=W2VK6RAQ6Y7FNG5SE7KWS4OWA2EYZJ54GQYKCFO3SNFVYMUG63YA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: <nil>
Mar 18 21:50:41.669663716 (197h39m44.685333317s) [seg_pos={0 0}, upload=189828]: Satisfied cohort requirements after 13 pieces
Mar 18 21:50:41.669986480 (197h39m44.685656089s) [noise=true , address=145.79.227.31:4798 , piece_id=S7DZDVOFQZKH52BV2WSXNYFKUXEKA47QFUUX3P33OUBIYHAMIRBA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF
Mar 18 21:50:41.669995260 (197h39m44.68566486s) [noise=true , address=145.79.227.31:4798 , piece_id=S7DZDVOFQZKH52BV2WSXNYFKUXEKA47QFUUX3P33OUBIYHAMIRBA, seg_pos={0 0}, upload=189828]: Long tail cancellation due to optimal threshold
Mar 18 21:50:41.670024179 (197h39m44.685693797s) [noise=true , address=77.90.31.18:4808 , piece_id=R2OX6UA26PPFISSE22DFGV6CIZMDR7Z4QJ5UXHBPFLSWQD6SECOQ, seg_pos={0 0}, upload=189828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Mar 18 21:50:41.670037817 (197h39m44.685707426s) [noise=true , address=77.90.31.18:4808 , piece_id=R2OX6UA26PPFISSE22DFGV6CIZMDR7Z4QJ5UXHBPFLSWQD6SECOQ, seg_pos={0 0}, upload=189828]: Long tail cancellation due to optimal threshold
Mar 18 21:50:41.670232411 (197h39m44.685902022s) [noise=true , address=79.127.219.41:4930 , piece_id=R7KNDLUIUAQUUTDGS3PU5CMTQUIFVVYIQU7CXVVIVGNJ57GQJTUA, seg_pos={0 0}, upload=189828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A39mM8Cc7tERvweTDoj4wkGgNTNn2QafEkMhn8PV1rwREj8WP): protocol: expected piece hash; context canceled; EOF
Mar 18 21:50:41.670243221 (197h39m44.685912831s) [noise=true , address=79.127.219.41:4930 , piece_id=R7KNDLUIUAQUUTDGS3PU5CMTQUIFVVYIQU7CXVVIVGNJ57GQJTUA, seg_pos={0 0}, upload=189828]: Long tail cancellation due to optimal threshold
Mar 18 21:50:41.670291176 (197h39m44.685960783s) [seg_pos={0 0}, upload=189828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:50:41.670399587 (197h39m44.6860692s) [upload=189828]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:50:41.670407742 (197h39m44.686077348s) [upload=189828]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:51:02.002389722 (197h40m5.018059345s) [upload=189828]: Done uploading segments...
Mar 18 21:51:02.002456155 (197h40m5.018125763s) [upload=189828]: Upload finished. err: <nil>
Duration: 30.003240435s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration:
5.00137958s trace:
https://jaeger.dev.storj.io/trace/60c64c4f23786ed8
FAIL (2026-03-18 21:50:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001402306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
12 out of 60 is failed (threshold=3) [.........................................XX....XXXXXX..XX.XX] duration:
30.005356537s trace:
https://jaeger.dev.storj.io/trace/4d8131add571fb64
FAIL (2026-03-18 21:50:56 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.005384803s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
8 out of 60 is failed (threshold=3) [...............................................XXX..XX.XX..X] duration:
30.00135678s trace:
https://jaeger.dev.storj.io/trace/2a0cc30dc4eb7cf7
FAIL (2026-03-18 21:50:55 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:50:25.628523850 (197h39m28.644193451s) [upload=189825]: Starting upload Mar 18 21:50:25.628582045 (197h39m28.644251634s) [upload=189825]: Done starting upload Mar 18 21:50:25.628664199 (197h39m28.644333788s) [upload=189825]: Uploading segments... Mar 18 21:50:25.628716752 (197h39m28.644386342s) [upload=189825]: Got next segment. Inline: true Mar 18 21:50:25.628722628 (197h39m28.644392228s) [upload=189825]: Waiting for error group managing segments... Mar 18 21:50:25.628769617 (197h39m28.644439208s) [upload=189825]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:50:25.628775162 (197h39m28.644444762s) [upload=189825]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:50:25.628778649 (197h39m28.644448248s) [upload=189825]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:50:46.266096655 (197h39m49.281766246s) [upload=189825]: Done uploading segments... Mar 18 21:50:46.266117592 (197h39m49.281787194s) [upload=189825]: Upload finished. err: <nil>
Duration: 30.001380878s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
48.089832724s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:50:43 UTC)
orbiter warning: context deadline exceeded
Duration: 48.089861146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................................X....X.XXXX.XXX..] duration:
22.944071055s trace:
https://jaeger.dev.storj.io/trace/11dc3d42043d3b39
FAIL (2026-03-18 21:50:31 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 22.944089644s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration:
5.001467783s trace:
https://jaeger.dev.storj.io/trace/6d606ff3c1dec74d
FAIL (2026-03-18 21:50:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0014965s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [........................................XX.....XXX..X..XXX.X] duration:
30.002587748s trace:
https://jaeger.dev.storj.io/trace/40000dbead1c6ff1
FAIL (2026-03-18 21:50:24 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:49:54.729674692 (197h38m57.745344303s) [upload=189820]: Starting upload
Mar 18 21:49:54.729779028 (197h38m57.745448654s) [upload=189820]: Done starting upload
Mar 18 21:49:54.729884587 (197h38m57.745554178s) [upload=189820]: Uploading segments...
Mar 18 21:49:54.730169512 (197h38m57.745839117s) [upload=189820]: Got next segment. Inline: false
Mar 18 21:49:54.730212358 (197h38m57.745881959s) [upload=189820]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:49:54.730219331 (197h38m57.745888932s) [upload=189820]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:49:59.000941337 (197h39m2.016610947s) [seg_pos={0 0}, upload=189820]: Begin upload segment...
Mar 18 21:49:59.001024661 (197h39m2.01669427s) [seg_pos={0 0}, upload=189820]: Done begin upload segment.
Mar 18 21:49:59.001139181 (197h39m2.016808792s) [noise=true , address=145.79.227.46:4605 , piece_id=6TEEMJ5ICGSJHWLUDWEEEUIFZLMTVATY4WJJRXKB3SMM3BQT7GZQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.002137662 (197h39m2.017807296s) [noise=true , address=145.79.227.22:4514 , piece_id=VZRJUNCLCCYNERWWVWMH2VQELGGEPBRMENDUBJSS54WTQIFTW5IA, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.003032612 (197h39m2.018702223s) [noise=true , address=145.79.227.17:4974 , piece_id=MSDS6FXLSKG4GPLEI6GN5XJGSZM3MJNWWLAL6UXUHO5N3NLKWGGQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.003594226 (197h39m2.019263836s) [noise=true , address=145.79.227.13:4806 , piece_id=NCS4MEL4HILNFCKMJYOOWMAH2Z4KBVMT2QYAOV5YDLEAFFMTWI2A, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.004361981 (197h39m2.020031587s) [noise=true , address=145.79.227.26:4657 , piece_id=X2F64ILVCS2HHDHCQHJQWV6TLEWHNJLNJBEMKULI5VOODLAEZ5NQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.004846847 (197h39m2.02051645s) [noise=true , address=145.79.227.5:4224 , piece_id=7H2SMD77J74GBV6V6TTGE4CCWCXRH64K5IEGXYF4YWMM3GHW5ZWQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.005292749 (197h39m2.020962356s) [noise=true , address=145.79.227.49:4338 , piece_id=Q7Y76CB3PHC3W62X3WNGJSSDGB4ODNTVX4N2VG4P26LOCCFFDPEA, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.005695681 (197h39m2.021365273s) [noise=true , address=145.79.227.23:4548 , piece_id=4CLWTCX6FHDGTGIDII3CTMTIZTAL7CJFMLGOYEXQNM7KMBIBJUSQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.007974271 (197h39m2.023643881s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.008691613 (197h39m2.024361222s) [noise=true , address=145.79.227.16:4915 , piece_id=TE7XEJRRETUJ5I4VVMKP2J7F3ZPTY2NWNTHNJLXSBXZF3UOTZKQA, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.010263764 (197h39m2.025933374s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.010892680 (197h39m2.02656229s) [noise=true , address=145.79.227.50:4375 , piece_id=XSRUK2PNASDBASH56XTL2GXXWVTDUFRHZCFXZHCBXGUJ6ZAM4JFQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.011500081 (197h39m2.027169693s) [noise=true , address=45.91.116.4:4939 , piece_id=NRM24OUWBKMLXL3K3JEUXHAK74YXZZ3HWIDUF26HKWANGMWPRAZA, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.013458920 (197h39m2.029128548s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.015443018 (197h39m2.031112629s) [noise=true , address=145.79.227.33:4870 , piece_id=OO4QWSZQR4DLX34TBT7MH7MGUYQVXPS4GU4QI5GN4OPBIHLPC3ZQ, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.017622563 (197h39m2.03329217s) [noise=true , address=45.91.116.12:4227 , piece_id=NZNDE5Y4BGU7OYHAXJU6TMNF2LM3PE4GJNKORAWRBY4HZFUKAP4A, seg_pos={0 0}, upload=189820]: Uploading piece...
Mar 18 21:49:59.615985686 (197h39m2.631655279s) [upload=189820]: Next returned nil segment
Mar 18 21:49:59.615999572 (197h39m2.631669176s) [upload=189820]: Waiting for error group managing segments...
Mar 18 21:49:59.748555653 (197h39m2.764225253s) [noise=true , address=145.79.227.17:4974 , piece_id=MSDS6FXLSKG4GPLEI6GN5XJGSZM3MJNWWLAL6UXUHO5N3NLKWGGQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.750995663 (197h39m2.766665263s) [noise=true , address=145.79.227.23:4548 , piece_id=4CLWTCX6FHDGTGIDII3CTMTIZTAL7CJFMLGOYEXQNM7KMBIBJUSQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.754590755 (197h39m2.770260357s) [noise=true , address=145.79.227.16:4915 , piece_id=TE7XEJRRETUJ5I4VVMKP2J7F3ZPTY2NWNTHNJLXSBXZF3UOTZKQA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.757716103 (197h39m2.773385704s) [noise=true , address=145.79.227.22:4514 , piece_id=VZRJUNCLCCYNERWWVWMH2VQELGGEPBRMENDUBJSS54WTQIFTW5IA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.757759155 (197h39m2.773428741s) [noise=true , address=145.79.227.5:4224 , piece_id=7H2SMD77J74GBV6V6TTGE4CCWCXRH64K5IEGXYF4YWMM3GHW5ZWQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.760731165 (197h39m2.776400783s) [noise=true , address=145.79.227.13:4806 , piece_id=NCS4MEL4HILNFCKMJYOOWMAH2Z4KBVMT2QYAOV5YDLEAFFMTWI2A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.761241065 (197h39m2.776910666s) [noise=true , address=145.79.227.26:4657 , piece_id=X2F64ILVCS2HHDHCQHJQWV6TLEWHNJLNJBEMKULI5VOODLAEZ5NQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.762447085 (197h39m2.778116677s) [noise=true , address=145.79.227.46:4605 , piece_id=6TEEMJ5ICGSJHWLUDWEEEUIFZLMTVATY4WJJRXKB3SMM3BQT7GZQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.763006585 (197h39m2.778676215s) [noise=true , address=145.79.227.50:4375 , piece_id=XSRUK2PNASDBASH56XTL2GXXWVTDUFRHZCFXZHCBXGUJ6ZAM4JFQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.763184247 (197h39m2.778853848s) [noise=true , address=45.91.116.12:4227 , piece_id=NZNDE5Y4BGU7OYHAXJU6TMNF2LM3PE4GJNKORAWRBY4HZFUKAP4A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.767249209 (197h39m2.782918826s) [noise=true , address=45.91.116.4:4939 , piece_id=NRM24OUWBKMLXL3K3JEUXHAK74YXZZ3HWIDUF26HKWANGMWPRAZA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.768646318 (197h39m2.78431592s) [noise=true , address=145.79.227.49:4338 , piece_id=Q7Y76CB3PHC3W62X3WNGJSSDGB4ODNTVX4N2VG4P26LOCCFFDPEA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.769381823 (197h39m2.785051415s) [noise=true , address=145.79.227.33:4870 , piece_id=OO4QWSZQR4DLX34TBT7MH7MGUYQVXPS4GU4QI5GN4OPBIHLPC3ZQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: <nil>
Mar 18 21:49:59.769397445 (197h39m2.785067046s) [seg_pos={0 0}, upload=189820]: Satisfied cohort requirements after 13 pieces
Mar 18 21:49:59.769620863 (197h39m2.78529047s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM): protocol: expected piece hash; context canceled; EOF
Mar 18 21:49:59.769627950 (197h39m2.785297551s) [noise=true , address=145.79.227.31:4801 , piece_id=WDDPUWK2XEYOUBDLAQTO5MLUF3ZPTDS5PQDJJL4MGYCKTUON2BXQ, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold
Mar 18 21:49:59.770067206 (197h39m2.785736816s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF
Mar 18 21:49:59.770078138 (197h39m2.785747742s) [noise=true , address=145.79.227.43:4160 , piece_id=FSJ3FSLWDLVJJO3ZO7ZPJ6TJKAWVO2G2IK27BQYSDBMOOSL2DTIA, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold
Mar 18 21:49:59.770220142 (197h39m2.785889773s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: expected piece hash; context canceled; EOF
Mar 18 21:49:59.770233071 (197h39m2.785902684s) [noise=true , address=145.79.227.45:4230 , piece_id=MG44Y3RPSJUMHDJ5OTVJH4UJUXTTPGSJZEP3O6UQBAHCAYWYLK3A, seg_pos={0 0}, upload=189820]: Long tail cancellation due to optimal threshold
Mar 18 21:49:59.770262306 (197h39m2.785931893s) [seg_pos={0 0}, upload=189820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:49:59.770377827 (197h39m2.786047438s) [upload=189820]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:49:59.770385336 (197h39m2.786054946s) [upload=189820]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:50:20.065891921 (197h39m23.081561528s) [upload=189820]: Done uploading segments...
Mar 18 21:50:20.065970376 (197h39m23.081639982s) [upload=189820]: Upload finished. err: <nil>
Duration: 30.002606203s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................................X....X.XXXX.XXX.] duration:
26.812748906s trace:
https://jaeger.dev.storj.io/trace/11dc3d42043d3b39
FAIL (2026-03-18 21:50:08 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 26.812781072s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
11 out of 60 is failed (threshold=3) [..........................................XX....XXXXXX..XX.X] duration:
30.003331458s trace:
https://jaeger.dev.storj.io/trace/22e91f00228cb631
FAIL (2026-03-18 21:50:02 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.003352273s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
7 out of 60 is failed (threshold=3) [................................................XXX..XX.XX..] duration:
29.846653241s trace:
https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5
FAIL (2026-03-18 21:49:55 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil>
Duration: 29.846692469s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........................................XX.....XXX..X..XXX.] duration:
27.19908997s trace:
https://jaeger.dev.storj.io/trace/251f894af766eb83
FAIL (2026-03-18 21:49:54 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:48:32.645171629 (197h37m35.660841237s) [upload=189796]: Starting upload
Mar 18 21:48:32.645326878 (197h37m35.660996481s) [upload=189796]: Done starting upload
Mar 18 21:48:32.645418130 (197h37m35.661087732s) [upload=189796]: Uploading segments...
Mar 18 21:48:32.645810289 (197h37m35.661479903s) [upload=189796]: Got next segment. Inline: false
Mar 18 21:48:32.645974442 (197h37m35.661644261s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:48:32.645981851 (197h37m35.661651455s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:48:45.066247463 (197h37m48.081917068s) [seg_pos={0 0}, upload=189796]: Begin upload segment...
Mar 18 21:48:45.066313671 (197h37m48.081983272s) [seg_pos={0 0}, upload=189796]: Done begin upload segment.
Mar 18 21:48:45.066379707 (197h37m48.082049312s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.066410147 (197h37m48.08207974s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.066971840 (197h37m48.082641444s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.067083170 (197h37m48.082752778s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.067554140 (197h37m48.083223754s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.068499259 (197h37m48.084168865s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.068748780 (197h37m48.084418385s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.069466321 (197h37m48.085135933s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.069654213 (197h37m48.085323824s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.070484066 (197h37m48.086153671s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.071245408 (197h37m48.086915016s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.071594691 (197h37m48.087264302s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.072277538 (197h37m48.087947139s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074170774 (197h37m48.08984036s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074261023 (197h37m48.089930611s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074812150 (197h37m48.090481742s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.668879318 (197h37m48.684548905s) [upload=189796]: Next returned nil segment
Mar 18 21:48:45.668896432 (197h37m48.684566039s) [upload=189796]: Waiting for error group managing segments...
Mar 18 21:48:45.797763739 (197h37m48.81343336s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.805326384 (197h37m48.820995984s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.806217578 (197h37m48.821887189s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.809363726 (197h37m48.825033341s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.811971344 (197h37m48.827640944s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.812960498 (197h37m48.828630103s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.813854402 (197h37m48.829524002s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.814801800 (197h37m48.830471401s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.823306084 (197h37m48.838975702s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.823905178 (197h37m48.839574778s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.824610241 (197h37m48.840279827s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.824966367 (197h37m48.840635966s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.831671478 (197h37m48.847341091s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.831697992 (197h37m48.847367609s) [seg_pos={0 0}, upload=189796]: Satisfied cohort requirements after 13 pieces
Mar 18 21:48:45.831975091 (197h37m48.8476447s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pvLo4F5ieDtepopg8Wi8jw6PKNeDCDvtH5iVNQXBUQwSXYxuD): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.831986640 (197h37m48.847656251s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837502196 (197h37m48.85317181s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.837522368 (197h37m48.853191978s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837711338 (197h37m48.853380951s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.837720674 (197h37m48.853390286s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837745572 (197h37m48.853415159s) [seg_pos={0 0}, upload=189796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:48:45.837833743 (197h37m48.853503354s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:48:45.837839430 (197h37m48.853509038s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:48:54.383402636 (197h37m57.399072251s) [upload=189796]: Done uploading segments...
Mar 18 21:48:54.383428351 (197h37m57.399097962s) [upload=189796]: Upload finished. err: <nil>
Duration: 27.199114115s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
27.770362689s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:49:52 UTC)
orbiter warning: context deadline exceeded
Duration: 27.770379656s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration:
5.00094823s trace:
https://jaeger.dev.storj.io/trace/2a1ec0d7f3bca161
FAIL (2026-03-18 21:49:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000971822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
40.110626321s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:49:25 UTC)
orbiter warning: context deadline exceeded
Duration: 40.110646606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
7 out of 60 is failed (threshold=3) [.................................................XXX..XX.XX.] duration:
21.787423914s trace:
https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5
FAIL (2026-03-18 21:49:24 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil>
Duration: 21.78745569s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................................XX.....XXX..X..XXX] duration:
30.003427132s trace:
https://jaeger.dev.storj.io/trace/251f894af766eb83
FAIL (2026-03-18 21:49:02 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:48:32.645171629 (197h37m35.660841237s) [upload=189796]: Starting upload
Mar 18 21:48:32.645326878 (197h37m35.660996481s) [upload=189796]: Done starting upload
Mar 18 21:48:32.645418130 (197h37m35.661087732s) [upload=189796]: Uploading segments...
Mar 18 21:48:32.645810289 (197h37m35.661479903s) [upload=189796]: Got next segment. Inline: false
Mar 18 21:48:32.645974442 (197h37m35.661644261s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:48:32.645981851 (197h37m35.661651455s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:48:45.066247463 (197h37m48.081917068s) [seg_pos={0 0}, upload=189796]: Begin upload segment...
Mar 18 21:48:45.066313671 (197h37m48.081983272s) [seg_pos={0 0}, upload=189796]: Done begin upload segment.
Mar 18 21:48:45.066379707 (197h37m48.082049312s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.066410147 (197h37m48.08207974s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.066971840 (197h37m48.082641444s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.067083170 (197h37m48.082752778s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.067554140 (197h37m48.083223754s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.068499259 (197h37m48.084168865s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.068748780 (197h37m48.084418385s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.069466321 (197h37m48.085135933s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.069654213 (197h37m48.085323824s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.070484066 (197h37m48.086153671s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.071245408 (197h37m48.086915016s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.071594691 (197h37m48.087264302s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.072277538 (197h37m48.087947139s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074170774 (197h37m48.08984036s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074261023 (197h37m48.089930611s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.074812150 (197h37m48.090481742s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Uploading piece...
Mar 18 21:48:45.668879318 (197h37m48.684548905s) [upload=189796]: Next returned nil segment
Mar 18 21:48:45.668896432 (197h37m48.684566039s) [upload=189796]: Waiting for error group managing segments...
Mar 18 21:48:45.797763739 (197h37m48.81343336s) [noise=true , address=77.90.31.13:4626 , piece_id=UCCXH7M3CDIRIKKGFJKQCQT4RFEFHSZPI7NKBXYFP3CANU3PBWPQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.805326384 (197h37m48.820995984s) [noise=true , address=79.127.219.38:4819 , piece_id=LRIIDEIEXD4DMBBA3QKZHHDQQHOQODJKEJN66TU7AHZJVSAAVBIA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.806217578 (197h37m48.821887189s) [noise=true , address=77.90.31.19:4845 , piece_id=PQBLNKD6T4LNJ5T4PVBA5X3XF5HWW4YTPQKSSTY2XDF7L4FWVHIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.809363726 (197h37m48.825033341s) [noise=true , address=145.79.227.27:4693 , piece_id=AMAIL7I4WUM4AAE24DBG3BRMGXX7WEUQZATQUXQEKY3ZTIGZVR7Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.811971344 (197h37m48.827640944s) [noise=true , address=145.79.227.2:4115 , piece_id=WFWUX4KGJK2L6TTMQBULRS432XOPQ57RYBUL27EH3I3XNM5XSLIQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.812960498 (197h37m48.828630103s) [noise=true , address=145.79.227.45:4234 , piece_id=2UU2KXZBUGG76M3GP54KY5DQSJJVDNMTEFGSZPSDYK2FEIIWLQ2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.813854402 (197h37m48.829524002s) [noise=true , address=145.79.227.39:4018 , piece_id=WWMQ3UWBJW4OTMS6MN4RGL5IFO6S2O66C7BT7ZC2CZGGOENYOXVQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.814801800 (197h37m48.830471401s) [noise=true , address=145.79.227.49:4337 , piece_id=GTATQZ6VN3OHO66HC3ET3ZACEN36YGIUJCXZRLZAP2PTODTX4VGA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.823306084 (197h37m48.838975702s) [noise=true , address=145.79.227.29:4947 , piece_id=64MOYAAS32IAWKJYDTAXQJ53VMKIRFL6KSJ3D6L5IGWC44BXUB3Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.823905178 (197h37m48.839574778s) [noise=true , address=145.79.227.15:4877 , piece_id=YNNQFMIU3J3KBOIPKQWEAHBAVH5ZCBNX3ADUGT56IWOMNIUAPN5Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.824610241 (197h37m48.840279827s) [noise=true , address=145.79.227.20:4441 , piece_id=DY3OXONC5WDIBXD45LTAVFL3QFYLK7ZSYOOFBATMCI3G6MMVOQ4Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.824966367 (197h37m48.840635966s) [noise=true , address=145.79.227.47:4269 , piece_id=JQ4M5B3VALY53H2BPBEABAFHBV34NJYTMDWOJIM7MRPJIAD22T2Q, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.831671478 (197h37m48.847341091s) [noise=true , address=45.91.116.4:4972 , piece_id=FPOKZSKHQ3X4RAZJ4XCRKTX3U5TJDR5M2GXPW23WPLZ2KCVJ3TZA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: <nil>
Mar 18 21:48:45.831697992 (197h37m48.847367609s) [seg_pos={0 0}, upload=189796]: Satisfied cohort requirements after 13 pieces
Mar 18 21:48:45.831975091 (197h37m48.8476447s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pvLo4F5ieDtepopg8Wi8jw6PKNeDCDvtH5iVNQXBUQwSXYxuD): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.831986640 (197h37m48.847656251s) [noise=true , address=79.127.219.41:4928 , piece_id=ZFAH74H6TLSCLSM35TEZ34KWHJFRZ7WGFNKB33O4UGC35B2UKTYQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837502196 (197h37m48.85317181s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.837522368 (197h37m48.853191978s) [noise=true , address=145.79.227.8:4333 , piece_id=CHV6WBTSWHYC57RDB5P4W5GJJJC6IKW7C7VCNG7YZDB7HMOJJMOA, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837711338 (197h37m48.853380951s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Mar 18 21:48:45.837720674 (197h37m48.853390286s) [noise=true , address=45.91.116.12:4227 , piece_id=VKFK2PQW3SZYOVQFS5ZKAQHQDM6EQET3EOAZMCS3H2NNRJOBCVSQ, seg_pos={0 0}, upload=189796]: Long tail cancellation due to optimal threshold
Mar 18 21:48:45.837745572 (197h37m48.853415159s) [seg_pos={0 0}, upload=189796]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:48:45.837833743 (197h37m48.853503354s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:48:45.837839430 (197h37m48.853509038s) [upload=189796]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:48:54.383402636 (197h37m57.399072251s) [upload=189796]: Done uploading segments...
Mar 18 21:48:54.383428351 (197h37m57.399097962s) [upload=189796]: Upload finished. err: <nil>
Duration: 30.003455447s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............................................X....X.XXXX.XXX] duration:
30.005707553s trace:
https://jaeger.dev.storj.io/trace/11dc3d42043d3b39
FAIL (2026-03-18 21:49:01 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.00573699s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
35.583060045s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:48:44 UTC)
orbiter warning: context deadline exceeded
Duration: 35.583081095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [...........................................XX....XXXXXX..XX.] duration:
25.638183989s trace:
https://jaeger.dev.storj.io/trace/6dbd9885f9de107d
FAIL (2026-03-18 21:48:43 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 25.638202752s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration:
5.002030887s trace:
https://jaeger.dev.storj.io/trace/593ce046ea9a8247
FAIL (2026-03-18 21:48:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002066647s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
7 out of 60 is failed (threshold=3) [..................................................XXX..XX.XX] duration:
30.002564875s trace:
https://jaeger.dev.storj.io/trace/70cfc1f7cfaaf2a5
FAIL (2026-03-18 21:48:19 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:47:49.893611547 (197h36m52.909281147s) [upload=189787]: Starting upload Mar 18 21:47:49.893758947 (197h36m52.909428545s) [upload=189787]: Done starting upload Mar 18 21:47:49.893869935 (197h36m52.909539537s) [upload=189787]: Uploading segments... Mar 18 21:47:49.893987843 (197h36m52.909662991s) [upload=189787]: Got next segment. Inline: true Mar 18 21:47:49.894002890 (197h36m52.909672498s) [upload=189787]: Waiting for error group managing segments... Mar 18 21:47:49.894077809 (197h36m52.909747412s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:47:49.894085400 (197h36m52.909754991s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:47:49.894089096 (197h36m52.909758687s) [upload=189787]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:48:14.608012362 (197h37m17.623681971s) [upload=189787]: Done uploading segments... Mar 18 21:48:14.608038608 (197h37m17.623708214s) [upload=189787]: Upload finished. err: <nil>
Duration: 30.002590336s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
10 out of 60 is failed (threshold=3) [............................................XX....XXXXXX..XX] duration:
30.001857974s trace:
https://jaeger.dev.storj.io/trace/6dbd9885f9de107d
FAIL (2026-03-18 21:48:18 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.001872098s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..............................................X....X.XXXX.XX] duration:
30.116176448s trace:
https://jaeger.dev.storj.io/trace/2bc345e63b93f9dd
FAIL (2026-03-18 21:48:09 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 30.116205418s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
51.36761058s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:48:09 UTC)
orbiter warning: context deadline exceeded
Duration: 51.367641032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...........................................XX.....XXX..X..XX] duration:
30.003642955s trace:
https://jaeger.dev.storj.io/trace/44d224acaa004e0
FAIL (2026-03-18 21:47:48 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:47:18.442808833 (197h36m21.458478442s) [upload=189779]: Starting upload
Mar 18 21:47:18.442901121 (197h36m21.458570723s) [upload=189779]: Done starting upload
Mar 18 21:47:18.442976117 (197h36m21.458645717s) [upload=189779]: Uploading segments...
Mar 18 21:47:18.443606581 (197h36m21.459276181s) [upload=189779]: Got next segment. Inline: false
Mar 18 21:47:18.443680474 (197h36m21.459350077s) [upload=189779]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:47:18.443699000 (197h36m21.459368603s) [upload=189779]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:47:26.699422791 (197h36m29.715092406s) [seg_pos={0 0}, upload=189779]: Begin upload segment...
Mar 18 21:47:26.699491248 (197h36m29.715160841s) [seg_pos={0 0}, upload=189779]: Done begin upload segment.
Mar 18 21:47:26.699526331 (197h36m29.715195943s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.702077766 (197h36m29.717747375s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.702861330 (197h36m29.71853093s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.702891630 (197h36m29.718561232s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.703323314 (197h36m29.718992924s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.703352656 (197h36m29.71902226s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.703987436 (197h36m29.719657047s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.703997584 (197h36m29.719667195s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.704537437 (197h36m29.720207043s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.704977460 (197h36m29.720647062s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.706372766 (197h36m29.722042367s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.707008899 (197h36m29.722678497s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.707089684 (197h36m29.722759294s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.708051043 (197h36m29.723720636s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.709186798 (197h36m29.724856393s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:26.712310630 (197h36m29.727980233s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Uploading piece...
Mar 18 21:47:27.306615628 (197h36m30.322285255s) [upload=189779]: Next returned nil segment
Mar 18 21:47:27.306668889 (197h36m30.322338504s) [upload=189779]: Waiting for error group managing segments...
Mar 18 21:47:27.442419844 (197h36m30.458089445s) [noise=true , address=145.79.227.22:4512 , piece_id=EC3ECV7JKIEBVYI3FHDP5OOON2LEATTJU4QZI4GNJCHVYWJ63NPQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.442434040 (197h36m30.458103645s) [noise=true , address=145.79.227.2:4119 , piece_id=V2ALCIE4DQRILQXVJKTUT6KIYKOPVB7LLKAM6OPT2J6NRAT3XFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.444434343 (197h36m30.460103947s) [noise=true , address=45.91.116.7:4019 , piece_id=KFQ5SSW7PG7OIKHVJYKCTIP7UGYND4KBSUTGX5LIEEWTMHTGDKCQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.445531395 (197h36m30.46120101s) [noise=true , address=145.79.227.23:4548 , piece_id=JN4PD6BWQMLI3776EWGRLEYKLDBZ7OEHVJYFFRQ6XXBZTU6W42FQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.445728504 (197h36m30.461398114s) [noise=true , address=145.79.227.30:4767 , piece_id=J4KYJJ3RGNIK6DZGZ75ABQQ2RKJYW7SNXBVR2DPUWKQXMTVTB3ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.448910049 (197h36m30.46457967s) [noise=true , address=45.91.116.4:4955 , piece_id=QGJEIRM2RYIH76L4FRWPPFAGB5GXIBQADBP722MY62COFMLQUQJQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.451992678 (197h36m30.467662287s) [noise=true , address=45.91.116.9:4296 , piece_id=DFVOW2WXMKHOA3WAHSIEDMFSAVXP3YSMUS6BRXARGWJ2OB7ZTD7A, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.452128814 (197h36m30.467798432s) [noise=true , address=45.91.116.10:4082 , piece_id=AJBTS3EUU35TQNBDYQ27VYCZS4HW6WCQUHFBXJ7KLOSFZGEC4VYQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.454127332 (197h36m30.469796945s) [noise=true , address=145.79.227.45:4235 , piece_id=67V22B23ABLKO75XQ6TBE25NLE463GPWY6KMH4UCHOYLXHDZJXSA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.455872225 (197h36m30.471541839s) [noise=true , address=145.79.227.19:4080 , piece_id=37V74ZR3HETTPLIB3DOGM5P4QBBINAVN3L2HJMPJ6PRNHMLUQ7IA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.458476641 (197h36m30.474146241s) [noise=true , address=145.79.227.31:4800 , piece_id=3B364YCKE6HGSLJOC4ZNRCBV6QUYNOBPCJHJV7XCIODROJPHMFWA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.461198518 (197h36m30.47686815s) [noise=true , address=145.79.227.8:4334 , piece_id=GIIWJEPHP5NPFBOOR3DRV7HJQNEPNI7G44MF65ZQDXDHSI2DV2ZQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.466630469 (197h36m30.482300072s) [noise=true , address=145.79.227.16:4911 , piece_id=Q442RAQ5D7OEM3XCA7OBXBMSBIOKCFZTP2QDUINOWB4VYPOYT5WQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: <nil>
Mar 18 21:47:27.466653132 (197h36m30.482322737s) [seg_pos={0 0}, upload=189779]: Satisfied cohort requirements after 13 pieces
Mar 18 21:47:27.466926351 (197h36m30.482595955s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL): protocol: expected piece hash; context canceled; EOF
Mar 18 21:47:27.466933928 (197h36m30.482603527s) [noise=true , address=145.79.227.12:4771 , piece_id=GRRFRSBOT7J4IYGWKWWVK4YGOI2IC7CRIZ4TNFGSQZ2QCPTUIELA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold
Mar 18 21:47:27.466972928 (197h36m30.482642554s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Mar 18 21:47:27.466988039 (197h36m30.482657651s) [noise=true , address=77.90.31.14:4662 , piece_id=QMEAIO3YDENQ73CZRC66D7O6IEBS4SE7QGJCWS6C6VIBO56IQ2QA, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold
Mar 18 21:47:27.467169807 (197h36m30.482839418s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi): protocol: expected piece hash; context canceled; EOF
Mar 18 21:47:27.467179258 (197h36m30.482848868s) [noise=true , address=145.79.227.44:4197 , piece_id=YKL7OSKMDB3MQAYNB44PQCWPCXZD4VVCQEDI3J32NJCIKDCFR2CQ, seg_pos={0 0}, upload=189779]: Long tail cancellation due to optimal threshold
Mar 18 21:47:27.467211812 (197h36m30.482881442s) [seg_pos={0 0}, upload=189779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:47:27.467312886 (197h36m30.482982494s) [upload=189779]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:47:27.467320096 (197h36m30.482989711s) [upload=189779]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:47:35.897519767 (197h36m38.913189375s) [upload=189779]: Done uploading segments...
Mar 18 21:47:35.897549016 (197h36m38.91321862s) [upload=189779]: Upload finished. err: <nil>
Duration: 30.003669693s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration:
5.000666722s trace:
https://jaeger.dev.storj.io/trace/549c36531950e590
FAIL (2026-03-18 21:47:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000823006s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...............................................X....X.XXXX.X] duration:
30.004707963s trace:
https://jaeger.dev.storj.io/trace/1799e0deebe9782a
FAIL (2026-03-18 21:47:27 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.004736265s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
9 out of 60 is failed (threshold=3) [.............................................XX....XXXXXX..X] duration:
30.001842368s trace:
https://jaeger.dev.storj.io/trace/f51217a9a46a2f1
FAIL (2026-03-18 21:47:22 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.001856253s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
6 out of 60 is failed (threshold=3) [...................................................XXX..XX.X] duration:
30.001896226s trace:
https://jaeger.dev.storj.io/trace/79f59160b77ec97a
FAIL (2026-03-18 21:47:13 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:46:44.213562861 (197h35m47.229232461s) [upload=189770]: Starting upload Mar 18 21:46:44.213634624 (197h35m47.229304213s) [upload=189770]: Done starting upload Mar 18 21:46:44.213688107 (197h35m47.229357718s) [upload=189770]: Uploading segments... Mar 18 21:46:44.213783933 (197h35m47.229453544s) [upload=189770]: Got next segment. Inline: true Mar 18 21:46:44.213793186 (197h35m47.229462797s) [upload=189770]: Waiting for error group managing segments... Mar 18 21:46:44.213873893 (197h35m47.229543493s) [upload=189770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:46:44.213881750 (197h35m47.229551356s) [upload=189770]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:46:44.213887721 (197h35m47.229557323s) [upload=189770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:46:56.859337880 (197h35m59.875007483s) [upload=189770]: Done uploading segments... Mar 18 21:46:56.859357249 (197h35m59.875026849s) [upload=189770]: Upload finished. err: <nil>
Duration: 30.001921231s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXXX] duration:
5.001462524s trace:
https://jaeger.dev.storj.io/trace/6f36e79e78a01382
FAIL (2026-03-18 21:47:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001491533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [............................................XX.....XXX..X..X] duration:
30.002391262s trace:
https://jaeger.dev.storj.io/trace/3deeefe5cb22d838
FAIL (2026-03-18 21:47:02 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:46:33.016860025 (197h35m36.032529635s) [upload=189768]: Starting upload
Mar 18 21:46:33.016989456 (197h35m36.032659062s) [upload=189768]: Done starting upload
Mar 18 21:46:33.017052165 (197h35m36.03272177s) [upload=189768]: Uploading segments...
Mar 18 21:46:33.017475356 (197h35m36.033144957s) [upload=189768]: Got next segment. Inline: false
Mar 18 21:46:33.017539668 (197h35m36.03320927s) [upload=189768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:46:33.017546318 (197h35m36.03321592s) [upload=189768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:46:37.290282605 (197h35m40.305952212s) [seg_pos={0 0}, upload=189768]: Begin upload segment...
Mar 18 21:46:37.290387539 (197h35m40.30605714s) [seg_pos={0 0}, upload=189768]: Done begin upload segment.
Mar 18 21:46:37.290453240 (197h35m40.306122851s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.290923120 (197h35m40.30659273s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.291701810 (197h35m40.307371663s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.292246979 (197h35m40.30791658s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.292332991 (197h35m40.308002591s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.292727864 (197h35m40.308397475s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.292767367 (197h35m40.308436977s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.293090621 (197h35m40.308760221s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.293273873 (197h35m40.308943484s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.293460336 (197h35m40.309129947s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.293769369 (197h35m40.309438976s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.293841511 (197h35m40.309511116s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.294279896 (197h35m40.309949487s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.294826369 (197h35m40.310495979s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.295420929 (197h35m40.311090529s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.295975630 (197h35m40.311645237s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Uploading piece...
Mar 18 21:46:37.895634454 (197h35m40.911304055s) [upload=189768]: Next returned nil segment
Mar 18 21:46:37.895651552 (197h35m40.911321163s) [upload=189768]: Waiting for error group managing segments...
Mar 18 21:46:38.034797744 (197h35m41.05046737s) [noise=true , address=45.91.116.10:4081 , piece_id=TG4WV6TPEEEHYIKKJRNKJ4MFCWIRERYIPVEWDKM3MXXS2FEULUMA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.034962121 (197h35m41.050631723s) [noise=true , address=145.79.227.35:4482 , piece_id=VP5MPTM7NWE2GYZAMO23SLT4JOXECXIUVEMCRHGBW4OTCXE4C4NQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.035419184 (197h35m41.051088784s) [noise=true , address=145.79.227.28:4731 , piece_id=TSU6F3CFW6ZDY5TKTZBP367ZMWKYMYLXL5RAWLJ65SKTTPN24XOA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.036603154 (197h35m41.052272755s) [noise=true , address=145.79.227.2:4115 , piece_id=HBNHXAJJW7ZBHIC7QR5WJ2AHRU6Y7KRTNIELZ6VRUGQXEQ4UWBXA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.036707110 (197h35m41.052376712s) [noise=true , address=145.79.227.41:4087 , piece_id=35AC6A2O3YFYZBA42BBMERFCOGGXVK3SWGD4EZOVA5KBRRHQFLKA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.037782694 (197h35m41.053452295s) [noise=true , address=77.90.31.11:4555 , piece_id=PTWBMPKHAQGZGG4NSR3ROQI2PVVWD6JHU4QZY3UAQWX5HFKREYIQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.039517974 (197h35m41.055187575s) [noise=true , address=145.79.227.21:4478 , piece_id=UQRH67W5TTO3VEOSB3GMVLLLOVQ6VYH6UB2S5AKEAJ7SFKAY3VVQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.039929620 (197h35m41.055599222s) [noise=true , address=145.79.227.39:4019 , piece_id=PLYDXCEW6ARWGBMCY7CBJODMNFKIAZEVNYKAS77YDV5WM7IJDDBA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.040264640 (197h35m41.055934243s) [noise=true , address=145.79.227.27:4694 , piece_id=S3QHVM4EKLSPG46JWPFB4LZVY2U67O437X6GIDESTXG6VL65U7RQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.040742968 (197h35m41.056412577s) [noise=true , address=145.79.227.30:4761 , piece_id=UBI7MD5CEYQJ52GLSB6ASUFSNQXZSP56ZWALIOAUS2UMVONWY4TA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.051123109 (197h35m41.066792713s) [noise=true , address=45.91.116.8:4045 , piece_id=XPQPGBYDPXRHFM3OPY6YAXPN6K4VWMTJD6B4PTLABILYYKJ2EJAA, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.056784301 (197h35m41.072453905s) [noise=true , address=45.91.116.7:4024 , piece_id=2K57RDMF24PP4KBYFGJZ4M5I4FAHREDTNGOZEQKE3KAIHDVRKJGQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.082796373 (197h35m41.098465981s) [noise=true , address=79.127.219.38:4821 , piece_id=A7OFPGVTWO76HIKOICEWYKVHPKSSEP4MV77MU3F7SOPU5UPODI3Q, seg_pos={0 0}, upload=189768]: Done uploading piece. err: <nil>
Mar 18 21:46:38.082828521 (197h35m41.098498126s) [seg_pos={0 0}, upload=189768]: Satisfied cohort requirements after 13 pieces
Mar 18 21:46:38.083138766 (197h35m41.098808369s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W4YnnqptDeH7t6QVMo988zAyAZ3Kj3dFE1yaxjttuWjrMvKeM): protocol: expected piece hash; context canceled; EOF
Mar 18 21:46:38.083150948 (197h35m41.098820558s) [noise=true , address=77.90.31.21:4881 , piece_id=57CPM65JNXQC5FV7E3TGTIIWYG2XJ6PQ3EP3CONSJN25BFHGJ4EQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold
Mar 18 21:46:38.083217334 (197h35m41.098886945s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Mar 18 21:46:38.083248682 (197h35m41.098918292s) [noise=true , address=77.90.31.19:4841 , piece_id=GYWGXSTMGCMB4R5X7FQOFTGFF7HWYYB3NY5ZVP23LD66BM57A3LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold
Mar 18 21:46:38.083440350 (197h35m41.099109956s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz): protocol: expected piece hash; context canceled; EOF
Mar 18 21:46:38.083448598 (197h35m41.099118197s) [noise=true , address=77.90.31.18:4809 , piece_id=NIROYHPGWV7Z7CR3PSRHO2NICBC4S4Z6YVRS4VWGJ7LUWBMRF4LQ, seg_pos={0 0}, upload=189768]: Long tail cancellation due to optimal threshold
Mar 18 21:46:38.083471664 (197h35m41.09914126s) [seg_pos={0 0}, upload=189768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:46:38.083578356 (197h35m41.099247957s) [upload=189768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:46:38.083588526 (197h35m41.099258126s) [upload=189768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:46:54.701340920 (197h35m57.717010532s) [upload=189768]: Done uploading segments...
Mar 18 21:46:54.701366952 (197h35m57.717036561s) [upload=189768]: Upload finished. err: <nil>
Duration: 30.002419338s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
8 out of 60 is failed (threshold=3) [..............................................XX....XXXXXX..] duration:
21.708301964s trace:
https://jaeger.dev.storj.io/trace/1c6f491aba147b1
FAIL (2026-03-18 21:46:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 21.708319921s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
32.267539216s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:46:39 UTC)
orbiter warning: context deadline exceeded
Duration: 32.267572524s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [................................................X....X.XXXX.] duration:
26.620919711s trace:
https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96
FAIL (2026-03-18 21:46:31 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 26.621007176s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
5 out of 60 is failed (threshold=3) [....................................................XXX..XX.] duration:
21.726106356s trace:
https://jaeger.dev.storj.io/trace/60f9f95a8f70aa89
FAIL (2026-03-18 21:46:15 UTC)
orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:44:41.998883275 (197h33m45.014552876s) [upload=189740]: Starting upload Mar 18 21:44:41.998978390 (197h33m45.014647999s) [upload=189740]: Done starting upload Mar 18 21:44:41.999110573 (197h33m45.014780159s) [upload=189740]: Uploading segments... Mar 18 21:44:41.999179787 (197h33m45.014849399s) [upload=189740]: Got next segment. Inline: true Mar 18 21:44:41.999187807 (197h33m45.014857418s) [upload=189740]: Waiting for error group managing segments... Mar 18 21:44:41.999255131 (197h33m45.014924733s) [upload=189740]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 18 21:44:41.999261855 (197h33m45.014931456s) [upload=189740]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 18 21:44:41.999266546 (197h33m45.014936148s) [upload=189740]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 18 21:45:10.708238593 (197h34m13.723908194s) [upload=189740]: Done uploading segments... Mar 18 21:45:10.708262832 (197h34m13.723932437s) [upload=189740]: Upload finished. err: <nil>
Duration: 21.726220195s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXXX] duration:
5.002293216s trace:
https://jaeger.dev.storj.io/trace/78d2ae33cae77162
FAIL (2026-03-18 21:46:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002342162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
47.828840268s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:46:06 UTC)
orbiter warning: context deadline exceeded
Duration: 47.828854707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)antiblip:
8 out of 60 is failed (threshold=3) [...............................................XX....XXXXXX.] duration:
26.034862831s trace:
https://jaeger.dev.storj.io/trace/1c6f491aba147b1
FAIL (2026-03-18 21:46:02 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 26.034895855s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............................................XX.....XXX..X..] duration:
22.959785305s trace:
https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb
FAIL (2026-03-18 21:46:01 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload
Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload
Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments...
Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false
Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment...
Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment.
Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment
Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments...
Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces
Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments...
Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil>
Duration: 22.959816925s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................................................X....X.XXXX] duration:
30.004684283s trace:
https://jaeger.dev.storj.io/trace/6d2ed2bf5cf64b96
FAIL (2026-03-18 21:45:58 UTC)
orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded
Duration: 30.00472724s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXXXXX] duration:
5.00101487s trace:
https://jaeger.dev.storj.io/trace/332ada5a7badd8c6
FAIL (2026-03-18 21:45:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001043189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
36.182926396s trace:
https://jaeger.dev.storj.io/trace/ebcefeaecc8948f
FAIL (2026-03-18 21:45:18 UTC)
orbiter warning: context deadline exceeded
Duration: 36.182954232s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............................................XX.....XXX..X.] duration:
27.136646187s trace:
https://jaeger.dev.storj.io/trace/3af580e4c9fbb8bb
FAIL (2026-03-18 21:45:11 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Mar 18 21:44:14.822177704 (197h33m17.837847304s) [upload=189734]: Starting upload
Mar 18 21:44:14.822236289 (197h33m17.837905891s) [upload=189734]: Done starting upload
Mar 18 21:44:14.822342546 (197h33m17.838012137s) [upload=189734]: Uploading segments...
Mar 18 21:44:14.822584492 (197h33m17.838254079s) [upload=189734]: Got next segment. Inline: false
Mar 18 21:44:14.822652539 (197h33m17.838322142s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 21:44:14.822658806 (197h33m17.838328418s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 21:44:31.107716166 (197h33m34.123385774s) [seg_pos={0 0}, upload=189734]: Begin upload segment...
Mar 18 21:44:31.108065288 (197h33m34.123734897s) [seg_pos={0 0}, upload=189734]: Done begin upload segment.
Mar 18 21:44:31.108176977 (197h33m34.12384657s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.108266034 (197h33m34.123935637s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.109216771 (197h33m34.12488638s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.109421694 (197h33m34.125091302s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110201647 (197h33m34.125871258s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110405328 (197h33m34.126074937s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110774370 (197h33m34.126443975s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.110812639 (197h33m34.126482251s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111238289 (197h33m34.126907899s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111263388 (197h33m34.126932997s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111686206 (197h33m34.127355816s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.111694878 (197h33m34.127364486s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112143129 (197h33m34.127812735s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112472505 (197h33m34.128142105s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.112788040 (197h33m34.128457643s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.113400877 (197h33m34.129070483s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Uploading piece...
Mar 18 21:44:31.718621806 (197h33m34.734291403s) [upload=189734]: Next returned nil segment
Mar 18 21:44:31.718638817 (197h33m34.734308436s) [upload=189734]: Waiting for error group managing segments...
Mar 18 21:44:31.857447213 (197h33m34.873116825s) [noise=true , address=145.79.227.9:4371 , piece_id=PONPXJM2S4MO7MYZCUCY7HPHO54E4M5GURCG5TU2XRUSCZQC6KJA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.857544663 (197h33m34.873214263s) [noise=true , address=45.91.116.14:4193 , piece_id=4YMXNIBCQPJ3NEFCV34G3EHZOXK35W3EELWFAUQGNHPOVPULYBMQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858439572 (197h33m34.874109175s) [noise=true , address=145.79.227.46:4601 , piece_id=YFD5ZPPXYIYHESHK2CQUFIS4UPCFS7LD3GWW3UWLZB6IHHO5I44Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858462478 (197h33m34.87413208s) [noise=true , address=145.79.227.45:4232 , piece_id=RMKYK5ZLCFLTEQPTE3QWAVBMQZS3UMOI4XQUUACQDKULVU6CY3LA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.858978335 (197h33m34.874647935s) [noise=true , address=45.91.116.7:4011 , piece_id=266ATW2AYMRIALKXDNLP3OFG6H6F5LNHYVMEWCR6EIWCRLWTYMHA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.862434240 (197h33m34.878103852s) [noise=true , address=145.79.227.26:4659 , piece_id=SU7EC4WIG3JGIEBNYFQWB4MNZSQOLCNGCF4GRS4EJNO2V2W7A3KA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.862544844 (197h33m34.878214444s) [noise=true , address=145.79.227.3:4154 , piece_id=3BZKCSE22PIN4DJ3ICSZ4JTSXAPIZBQSFOXP67YL7YBBQRKDQVSA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.863976492 (197h33m34.879646102s) [noise=true , address=145.79.227.51:4409 , piece_id=XCOLFCU5XIIIJGVM7P6YUM76OJPIHFMD3R34YSXFABQRCJLUZS4Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.864285028 (197h33m34.879954637s) [noise=true , address=145.79.227.13:4805 , piece_id=C23D7LMKJVMKKXKS6AMFQF3AKIXA6MWY475KGM7NLQ7WMAY4P67Q, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.865736653 (197h33m34.88140626s) [noise=true , address=145.79.227.35:4485 , piece_id=OVFBYPA2ZPNK5PLRZICHCX6OJ6K7HALLOBR4WCR7QH244RQ6NOSQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.867258649 (197h33m34.882928252s) [noise=true , address=145.79.227.2:4119 , piece_id=DCKKGYXALFOQO4L3QLEKBWDBDEHJLMUIS6RQP5CMGSZE54C244BQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.867691333 (197h33m34.883360937s) [noise=true , address=145.79.227.22:4511 , piece_id=3LL5EOKLVPNOGS3J5YOT2D77MI7KIQ24LTRZXCX5UUWFUGAEGIFA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.868441357 (197h33m34.884110958s) [noise=true , address=145.79.227.14:4843 , piece_id=NAC5OJNATEGWO3ULRT6UKTWRCOD2TGOA7I6D5BEP4GU2Z2S2X4UA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: <nil>
Mar 18 21:44:31.868455518 (197h33m34.884125112s) [seg_pos={0 0}, upload=189734]: Satisfied cohort requirements after 13 pieces
Mar 18 21:44:31.868828711 (197h33m34.88449832s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.868837407 (197h33m34.884506997s) [noise=true , address=145.79.227.48:4303 , piece_id=ZDFJ36RJJKDVUJYB6OVS5K2BXOF5EMJJNI5HSDQGNCTM3NMVCIGA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869175415 (197h33m34.884845024s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.869183221 (197h33m34.884852822s) [noise=true , address=145.79.227.19:4083 , piece_id=OYFTHUJ4XKKDT2DFXHGGQWQCOYIJQTHBSHEL2PPN7FPMJ4XZWSUA, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869573509 (197h33m34.885243118s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DcpXz1fKrWhdPxTpGUFmydMUuzMQpNFMgGB75BAVrjdMiSkhW): protocol: expected piece hash; context canceled; EOF
Mar 18 21:44:31.869599153 (197h33m34.885268753s) [noise=true , address=145.79.227.50:4378 , piece_id=JD4C4YWV27TLKDSWH4X3KZR5IVTYAUDO6T4AIUIR5QCDRVDXRUQQ, seg_pos={0 0}, upload=189734]: Long tail cancellation due to optimal threshold
Mar 18 21:44:31.869623374 (197h33m34.885292964s) [seg_pos={0 0}, upload=189734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Mar 18 21:44:31.869727986 (197h33m34.885397587s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 18 21:44:31.869735661 (197h33m34.885405261s) [upload=189734]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 18 21:44:40.476339100 (197h33m43.492008705s) [upload=189734]: Done uploading segments...
Mar 18 21:44:40.476371656 (197h33m43.492041256s) [upload=189734]: Upload finished. err: <nil>
Duration: 27.136676097s
core
uplink
us-select-1
|