Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................................................X.........] duration:
1.692164767s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:46:01 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.692200584s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................................................X........] duration:
1.715386429s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:45:39 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.715544505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................................................X.......] duration:
2.178046519s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:45:24 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 2.178117272s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................................................X......] duration:
1.608925096s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:44:40 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.608957003s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................................................X.....] duration:
1.647570696s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:44:24 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.647642705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................................................X....] duration:
2.154765476s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:43:19 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 2.154793735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
1.707545032s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:42:49 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.707636395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
1.698249122s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:42:15 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.698285197s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
1.66389155s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:41:34 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 1.663990825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.00084989s trace:
https://jaeger.dev.storj.io/trace/63576a9f35a3b8c0
FAIL (2026-04-06 20:41:11 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000876585s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
1.708761923s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:24:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.708797185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [XX..........................................................] duration:
1.754426529s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:24:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.754450616s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
3.047809393s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:24:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.047844393s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.543988011s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:23:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.544020553s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.XX.........................................................] duration:
1.700693264s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:23:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.700745994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.879085269s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:23:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.879115536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..XX........................................................] duration:
1.670266199s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:22:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.670312785s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.405571131s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:22:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.405610139s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
3.676393503s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:22:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.676425968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
2.801897803s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:22:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.801989119s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.985391357s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:21:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.985493105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
2.987885632s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:21:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.987911532s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...XX.......................................................] duration:
2.189275669s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:21:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.189388298s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.359871809s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:21:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.359912834s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
3.01072098s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:21:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.010749723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
2.473096206s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:21:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.473313127s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
2.686340013s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:20:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.686373648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
3.385674733s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:20:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.385710978s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....XX......................................................] duration:
1.547385966s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:20:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.547501409s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
3.122693192s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:20:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.12272552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.370334826s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:20:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.370368116s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....XX.....................................................] duration:
1.73996652s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:20:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.74001564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration:
2.834514748s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:20:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.834617706s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
2.513221477s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:19:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.513238623s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
3.042809032s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:19:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.042840758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
3.301080051s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:19:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.301104552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......XX....................................................] duration:
1.66137703s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:19:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.661406462s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
3.157710955s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:18:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.157742017s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
2.657682187s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:18:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.657772885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
2.324274159s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:18:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.324378966s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
3.283722472s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:18:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.283750461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
3.621036861s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:18:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.621154811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......XX...................................................] duration:
1.750895021s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:18:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.75092206s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
2.680369327s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:17:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.680394917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
2.373774769s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:17:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.373804625s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
3.126733875s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:17:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.126773354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........XX..................................................] duration:
1.547669441s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:17:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.547703221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
2.760017732s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:17:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.760043311s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
2.941196322s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:16:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.941227198s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
2.674171182s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:16:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.674204544s
core
uplink
us-select-1
us1-placement-20
|