Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
3.079974846s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:07:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.080005567s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
2.989876065s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:07:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.98990356s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.451507058s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:07:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.451612311s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................XX......................................] duration:
1.750042322s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:07:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.750117279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.535555078s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:06:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.535587273s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
3.107423998s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:06:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.107458829s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
3.393379414s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:06:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.393420543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................XX.....................................] duration:
2.180047779s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:06:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.180138128s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................XX....................................] duration:
2.137683558s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:05:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.137742123s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.375593s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:05:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.375623357s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
3.21842149s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:05:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.21853154s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................XX...................................] duration:
1.523659074s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:05:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.523696466s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
2.788712839s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:05:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.788750715s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................XX..................................] duration:
1.568814947s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:04:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.568857912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.375988635s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:04:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.376021282s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
3.236025855s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:04:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.23606331s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................XX.................................] duration:
1.570601825s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:04:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.570634984s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.345440633s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:04:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.345487844s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
2.580166809s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:03:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.58025658s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................XX................................] duration:
2.080592961s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:03:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.080620737s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
3.02304802s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:03:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.023082766s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
2.72639083s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:03:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.726437596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.41298271s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:03:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.413032045s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
3.025523476s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:02:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.025554648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................XX...............................] duration:
1.603397555s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:02:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.603431342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.35270429s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:02:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.352728405s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................XX............................] duration:
3.110245873s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:02:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.110272656s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................XX..............................] duration:
2.217778055s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:02:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.217810872s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.484583791s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:02:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.484614477s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
3.071462626s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:01:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.071482564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................XX...........................] duration:
2.666258601s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:01:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.666284422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.297152871s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:01:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.297183142s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................XX..........................] duration:
3.223784628s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:01:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.223811499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
3.060510914s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:01:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.060538237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................XX.............................] duration:
1.581410641s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:01:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.581505358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.377833465s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:00:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.377862704s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................XX.........................] duration:
3.341359459s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 16:00:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.341399283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..............................XX............................] duration:
1.649163318s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 16:00:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.64927263s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
3.078812876s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:00:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.078862604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.393515433s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 16:00:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.393682163s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
3.449737061s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 16:00:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.449841967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................XX........................] duration:
2.602343523s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:59:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.602376798s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...............................XX...........................] duration:
1.56891467s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:59:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.568954221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
3.768862738s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:59:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.768898515s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.276492676s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:59:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.276528635s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [................................XX..........................] duration:
1.585819926s trace:
https://jaeger.dev.storj.io/trace/3a14434996c4db35
FAIL (2026-04-06 15:59:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.58593619s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
3.046753334s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:58:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.046799421s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.279702855s trace:
https://jaeger.dev.storj.io/trace/24b7cbe3868ea029
FAIL (2026-04-06 15:58:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Apr 6 15:40:59.113200131 (647h30m2.128869732s) [upload=621886]: Starting upload
Apr 6 15:40:59.113290641 (647h30m2.128960241s) [upload=621886]: Done starting upload
Apr 6 15:40:59.113337138 (647h30m2.129006737s) [upload=621886]: Uploading segments...
Apr 6 15:40:59.113714459 (647h30m2.12938406s) [upload=621886]: Got next segment. Inline: false
Apr 6 15:40:59.113752306 (647h30m2.129421905s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:59.113757955 (647h30m2.129427556s) [upload=621886]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:59.227937557 (647h30m2.243607167s) [seg_pos={0 0}, upload=621886]: Begin upload segment...
Apr 6 15:40:59.228033538 (647h30m2.243703143s) [seg_pos={0 0}, upload=621886]: Done begin upload segment.
Apr 6 15:40:59.228103967 (647h30m2.243773578s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.228262422 (647h30m2.243932032s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229476673 (647h30m2.245146282s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.229718113 (647h30m2.245387723s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230365564 (647h30m2.246035175s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.230949202 (647h30m2.246618812s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.231495476 (647h30m2.247165086s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232048158 (647h30m2.247717768s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.232612357 (647h30m2.248281973s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.233171134 (647h30m2.248840746s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.234615191 (647h30m2.250284801s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.235641719 (647h30m2.25131133s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236130271 (647h30m2.251799878s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.236592379 (647h30m2.252261989s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237129334 (647h30m2.252798943s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.237618331 (647h30m2.253287942s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Uploading piece...
Apr 6 15:40:59.848643716 (647h30m2.864313305s) [upload=621886]: Next returned nil segment
Apr 6 15:40:59.848666658 (647h30m2.86433627s) [upload=621886]: Waiting for error group managing segments...
Apr 6 15:40:59.974661025 (647h30m2.990330629s) [noise=true , address=145.79.227.8:4335 , piece_id=GMEW5VVZU2T7A7JJHD55Y42BELN7GHELYKDS5HMXCZ2Z2ODYXKNQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.977168292 (647h30m2.992837897s) [noise=true , address=145.79.227.5:4221 , piece_id=CF7TZGIABSWVW6LMRB56C73YVNG3VNLDDRLIDKHASKR7QUDFZBQA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.980846415 (647h30m2.996516027s) [noise=true , address=145.79.227.9:4369 , piece_id=D6CQLVXPXLKI4T2HYHAMAIQOYCU5ZPGPQX6354XBT76HCOKY4PLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.985649355 (647h30m3.001318961s) [noise=true , address=145.79.227.26:4654 , piece_id=KN2AZUA2FVY2MIGGU2VVBQ3J5DCNDS65P5N5XI2KKAWMDP43CENQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.986739092 (647h30m3.002408692s) [noise=true , address=145.79.227.28:4728 , piece_id=7G2PIBQ63IYHR5QRY3I4P7L3PJD5FGFW7OS3SX43RB2MIMJEHPNA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.990327795 (647h30m3.005997394s) [noise=true , address=145.79.227.10:4407 , piece_id=KMGRPGXUV2NDWTO4AF6GP3C2PU6JRVJCBZTRFWRTQPF4D52M6QRQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995230737 (647h30m3.010900338s) [noise=true , address=77.90.31.10:4521 , piece_id=2GQHJWWWWQXS6ONYPQAADT35SD2VXASHD6QPPORDUQ4U5RRHNXEA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:40:59.995433521 (647h30m3.011103121s) [noise=true , address=145.79.227.49:4341 , piece_id=5U4BA3RNELXP5O2WHRBR3TGIVECARMBDC5TMO6KFTMQRFU2KMAHQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.009137822 (647h30m3.024807425s) [noise=true , address=145.79.227.36:4521 , piece_id=LZ4SXHOK3CPBX4F3MROFGFYU6DK2Q2DT37CQRQJPRWW4ST5QDL4A, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.013755800 (647h30m3.029425421s) [noise=true , address=145.79.227.47:4271 , piece_id=4BTQYDRMTS4HQO47UH7YTZNR6HAD67EAXXX2EAP7VPZBYCLTBIYQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:00.015419724 (647h30m3.031089331s) [noise=true , address=145.79.227.7:4298 , piece_id=W4DFL3YTIFLWXHUGD5AHBFH2KAYD6OXZQRPNIDBUSBONB4FZ3RCA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: <nil>
Apr 6 15:41:28.813097401 (647h30m31.828767021s) [noise=true , address=45.91.116.12:4225 , piece_id=LWUL2GB55F7SUQRF3R5RPASCTEVDSFRDJ766D3LEJY2CLHC63ZXA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 139.59.144.22:49794->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813155500 (647h30m31.82882511s) [noise=true , address=45.91.116.14:4191 , piece_id=NYIKIILPYKFOEWHZVFEPYGDUY5LLY77ONQTBUZSC5IRY36RVZBLA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 139.59.144.22:41748->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813349159 (647h30m31.829018773s) [noise=true , address=45.91.116.10:4084 , piece_id=AAYLK4EF7YTF5FY24EYUIMUYZDH5ZB323EHGAVEOGGHVW4UF7ESQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, address:45.91.116.10:4084): protocol: read tcp 139.59.144.22:40602->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813525379 (647h30m31.82919501s) [noise=true , address=45.91.116.7:4024 , piece_id=MDVTPODCR2DAMWTJQS5N3WUOI47TOARP2XO6FVNN6S2JRQ7DLWJA, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12PTKXHL7D3hiUs3jDLnV8wBCJgrgqdeYkyghP1h4wZCMezCaR, address:45.91.116.7:4024): protocol: read tcp 139.59.144.22:34542->45.91.116.7:4024: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813589353 (647h30m31.82925895s) [noise=true , address=45.91.116.5:4301 , piece_id=IFUTQA2JKHEAAECQUDLOXNYNT3RW2ALZ62ZXHAKT5JQLNDLMFKIQ, seg_pos={0 0}, upload=621886]: Done uploading piece. err: ecclient: upload failed (node:12WdUew8BjuMfHyBJ55HwsNFXuWRnBZeNhkCtdhotTgsjSx2Rpg, address:45.91.116.5:4301): protocol: read tcp 139.59.144.22:59900->45.91.116.5:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:28.813632781 (647h30m31.829302376s) [seg_pos={0 0}, upload=621886]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Apr 6 15:41:28.926471512 (647h30m31.942141122s) [upload=621886]: Done uploading segments...
Apr 6 15:41:28.926518632 (647h30m31.942188244s) [upload=621886]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.279735289s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................XX.......................] duration:
2.636308547s trace:
https://jaeger.dev.storj.io/trace/73f2db13275e353b
FAIL (2026-04-06 15:58:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.636335336s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
3.946447462s trace:
https://jaeger.dev.storj.io/trace/282f3f952d3c39b9
FAIL (2026-04-06 15:58:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
se
Apr 6 15:40:03.441484386 (647h29m6.457153979s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 6 15:40:03.441489531 (647h29m6.457159134s) [upload=621870]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 6 15:40:03.553469608 (647h29m6.569139222s) [seg_pos={0 0}, upload=621870]: Begin upload segment...
Apr 6 15:40:03.553567446 (647h29m6.569237067s) [seg_pos={0 0}, upload=621870]: Done begin upload segment.
Apr 6 15:40:03.553628274 (647h29m6.569297901s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555144703 (647h29m6.570814317s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.555821247 (647h29m6.571490855s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556341504 (647h29m6.572011118s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.556967796 (647h29m6.572637406s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557471121 (647h29m6.573140741s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.557987513 (647h29m6.573657125s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558483802 (647h29m6.574153419s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.558935721 (647h29m6.574605331s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559406769 (647h29m6.575076379s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.559878194 (647h29m6.575547802s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.560284715 (647h29m6.575954329s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.561580886 (647h29m6.577250501s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562168696 (647h29m6.577838309s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.562669214 (647h29m6.578338827s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563151475 (647h29m6.578821087s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.563587570 (647h29m6.579257184s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:03.564005547 (647h29m6.579675159s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Uploading piece...
Apr 6 15:40:04.141500404 (647h29m7.157170027s) [upload=621870]: Next returned nil segment
Apr 6 15:40:04.141522569 (647h29m7.15719219s) [upload=621870]: Waiting for error group managing segments...
Apr 6 15:40:04.304476951 (647h29m7.320146558s) [noise=true , address=77.90.31.15:4725 , piece_id=VI6OSDYYRO46VA5H6DTGNDFMFHV2YR3W6NM6VGVHFVTBVZXVSQQA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.321029537 (647h29m7.336699148s) [noise=true , address=77.90.31.21:4901 , piece_id=4HQ7WVAG5CSNISSNIMTHPGXS6UJZHV3YLGFLVTLST5AT6E3BTWQQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.342498739 (647h29m7.358168358s) [noise=true , address=77.90.31.16:4764 , piece_id=GJCUUOFV3ZJU2AZ2TPOE5MEWAOI3ASIMDXRL5G7NXLF6NSEYVMDA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.694622833 (647h29m7.710292453s) [noise=true , address=167.17.72.70:4768 , piece_id=RDZ63BNZQXF7GMGYX657BDZ4KOHE2AKUU2M2RP3PLQ64TU4Z3IWA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809275676 (647h29m7.824945279s) [noise=true , address=193.149.164.70:4121 , piece_id=BPVV7IOUP5CHD3RLFJSJ42HMX32HQQBYV2KSRKSDYXY2GO2VY43A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.809632477 (647h29m7.825302116s) [noise=true , address=193.149.164.66:4996 , piece_id=FFQNVW5QA7ULTWFYOAZLSRAV3HEZ3GNCZFRDWGCGO7ZDECBXWMBA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.814979498 (647h29m7.830649115s) [noise=true , address=193.149.164.67:4031 , piece_id=UCNEMJJ5BJG7ZB5UO37K65WWJCW6OMTH4ZKWONDE3KZB6EQ6NR3Q, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.820205102 (647h29m7.835874715s) [noise=true , address=193.149.164.68:4042 , piece_id=627ADQ4TYIXZJSBTF4EBJV3VABBQ5B2VJXDVJZKYX7OU5VYDWA4A, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.849152629 (647h29m7.864822251s) [noise=true , address=193.149.164.71:4318 , piece_id=JRCNCOHOUEGR7L2R3FHEVG6MHVY5SFBPLPJTIWIUH2FQ4GISV3CQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:04.879775032 (647h29m7.895444647s) [noise=true , address=193.149.164.69:4096 , piece_id=LWTYRPJD4ZQTLVRQUTF7UZCYEIAHV2VPYM56UFEM6WH4OKFFZTJA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.133558986 (647h29m8.149228589s) [noise=true , address=167.17.72.71:4802 , piece_id=3EFW2RW7IOA456GXAZSHPSTM3GTAEVSBG7PM2QOSDMU373Q4H7KQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:05.154919931 (647h29m8.170589541s) [noise=true , address=145.79.227.34:4919 , piece_id=Q66QF3VVWEYSEPVKTLKUMP6FNCGJ36PDWZTS6DQTHBJDL6APHYUA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:40:07.991118972 (647h29m11.006788579s) [noise=true , address=193.149.164.143:4702 , piece_id=CY32N3QDFHAQQGAZFKYCNTWKRAUOTY6W45D2ARQN35FZYATOA4FA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: <nil>
Apr 6 15:41:03.128365133 (647h30m6.144034801s) [noise=true , address=45.91.116.8:4064 , piece_id=X6CA5OICXGE6J3Y3EMVIEQHYD2LENWQT4KQQEX3647WNMJTOJCAQ, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:124guAM82oibYZsMho67Em6Ke6yEZXwT6LPMLLSZZBLqwoFGUkn, address:45.91.116.8:4064): protocol: read tcp 139.59.144.22:55076->45.91.116.8:4064: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128409784 (647h30m6.144079403s) [noise=true , address=45.91.116.12:4238 , piece_id=GOFVGN4YQAASX2SVPKBY2DYQAAYT5CA3WIYTPHVOPAHLWXUW6ACA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1yd47We7k2qgmcAVz5DhwZQMfnX71BrmLmtBoYyLn7D8t1psh2, address:45.91.116.12:4238): protocol: read tcp 139.59.144.22:39470->45.91.116.12:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128581967 (647h30m6.144251626s) [noise=true , address=45.91.116.11:4146 , piece_id=AROW6W6PTKV27KOQTL2R5VI3YKVTAFR6RW23IXIKUAFVJXURY2IA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: read tcp 139.59.144.22:53902->45.91.116.11:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.128928873 (647h30m6.144598492s) [noise=true , address=45.91.116.10:4104 , piece_id=MH5BZRGQLZBTEDTOEMTOTCZ6WOT2IQWULZO5GBPCA43XIUWSK5NA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 139.59.144.22:52722->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129037650 (647h30m6.144707259s) [noise=true , address=45.91.116.7:4041 , piece_id=TJ34L3TWFXW2IKGCJH2C4YUWH22AXEAVNG2ZOFEWDCMOBSDPKCFA, seg_pos={0 0}, upload=621870]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: read tcp 139.59.144.22:53046->45.91.116.7:4041: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 6 15:41:03.129088475 (647h30m6.144758077s) [seg_pos={0 0}, upload=621870]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Apr 6 15:41:03.235548245 (647h30m6.25121786s) [upload=621870]: Done uploading segments...
Apr 6 15:41:03.235598454 (647h30m6.251268075s) [upload=621870]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 3.946492983s
core
uplink
us-select-1
us1-placement-20
|