Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.37415457s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:58:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.374192165s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................XX.........................] duration:
2.075469949s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:58:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.075554693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................XX......................] duration:
2.695562936s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:57:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.695607266s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.35607398s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:57:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.356103207s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................XX........................] duration:
2.123636498s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:57:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.123758111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
3.084172185s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:57:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.084228124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.369124292s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:57:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.369148673s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration:
2.673967369s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:56:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.674110635s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
3.412984744s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:56:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.413026085s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................XX.......................] duration:
1.592957837s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:56:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.592990264s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................XX....................] duration:
2.707823582s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:56:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.707856858s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.391904021s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:56:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.391934032s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................XX......................] duration:
1.649843533s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:56:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.649869635s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
3.254464035s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:55:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.254493816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................XX...................] duration:
3.075348363s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:55:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.075373392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.459398362s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:55:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.45943008s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration:
1.7286481s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:55:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.7286805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
3.439889464s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:55:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.440001587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.503255458s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:55:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.503345086s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................XX..................] duration:
3.205067017s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:55:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.205112079s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................................XX....................] duration:
1.894921611s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:54:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.894968808s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................XX...................] duration:
1.758316193s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:54:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.758369828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
3.310604391s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:54:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.310699795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................XX.................] duration:
2.728209522s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:54:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.728245169s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
3.161653093s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:54:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.161684849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.496701082s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:54:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.496739214s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................XX..................] duration:
1.812393836s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:53:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.812477806s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................XX................] duration:
3.130894629s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:53:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.130949709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
3.31750365s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:53:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.317530929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.491428836s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:53:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.491514311s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
3.107992283s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:52:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.108019136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................XX.................] duration:
1.730008682s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:52:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.730033257s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................................XX...............] duration:
2.944964043s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:52:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.944993987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
3.58219518s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:52:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.582224985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................................XX................] duration:
2.22119174s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:52:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.221230338s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.340638424s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:52:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.340669954s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.381568624s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:52:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.381595592s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
3.156313552s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:51:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.156347119s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................................XX..............] duration:
2.972435232s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:51:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.972462387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................................XX...............] duration:
2.321184481s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:51:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.321248074s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.363202231s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:51:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.363232571s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
3.476280051s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:50:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.476311416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................................XX..............] duration:
1.83192261s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:50:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.831965766s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................................XX.............] duration:
2.831844769s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:50:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.831897198s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
3.334436629s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:50:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.334465391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.518781961s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:50:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.518815925s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
3.458186535s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:49:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.458279248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................................XX.............] duration:
2.418617713s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:49:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.418650304s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................................XX............] duration:
3.084397354s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:49:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.08455545s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................................XX............] duration:
2.187029344s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:49:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.187062336s
core
uplink
us-select-1
us1-placement-20
|