Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 185828 reports (Page 73 of 3717)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
2.341424907s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:54:26 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.34145717s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [.....X.........]
duration:
165.464735ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:54:15 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 165.489656ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
2.440098041s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:54:06 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.440149118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration:
2.438828441s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:53:47 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.438889548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................]
duration:
2.380098704s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:53:24 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.380124325s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [......X........]
duration:
163.47303ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:53:17 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 163.498209ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
2.572742634s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:52:32 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.572767404s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [.......X.......]
duration:
186.848048ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:52:20 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 186.878222ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration:
2.432289735s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:51:45 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.432315648s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [........X......]
duration:
157.419569ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:51:28 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 157.445602ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................]
duration:
2.388926207s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:50:44 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.388952461s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [.........X.....]
duration:
165.165812ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:50:36 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 165.191352ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................]
duration:
2.259857847s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:50:20 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.259885615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration:
2.324469387s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:49:53 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.324498418s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [..........X....]
duration:
163.901795ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:49:50 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 163.993605ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [...........X...]
duration:
170.604291ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:49:30 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 170.630994ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration:
2.28359372s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:49:00 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.283620814s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [............X..]
duration:
155.767401ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:48:33 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 155.792771ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................]
duration:
2.319924136s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:48:21 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.319952327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................]
duration:
2.516328765s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:47:36 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.516358868s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [.............X.]
duration:
184.035129ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:47:19 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 184.081406ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [..............X]
duration:
99.550284ms
remote:
"tcp:[2620:cb:2000::1]:443"
response headers:
Content-Type=text/plain; charset=UTF-8
Content-Length=15
Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires=Thu, 01 Jan 1970 00:00:01 GMT
Cf-Ray=9ade7e5d7aae9963-FRA
X-Frame-Options=SAMEORIGIN
Date=Sun, 14 Dec 2025 14:46:56 GMT
Referrer-Policy=same-origin
Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/1244b8592b06e8c8
FAIL (2025-12-14 14:46:56 UTC)
orbiter warning: unexpected status code "502 Bad Gateway"
"error code: 500"
Duration: 99.577018ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration:
2.4382214s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:46:47 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.43832894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration:
2.420817711s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:46:20 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.420848188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................]
duration:
2.3816382s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:45:22 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.381668039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration:
2.346881105s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:44:26 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.346910755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration:
2.331783963s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:43:14 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.331816542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration:
2.508647455s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:42:05 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.508678876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration:
2.484208804s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:41:12 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.484239362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................]
duration:
2.294311797s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:40:48 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.294340069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration:
2.329415289s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:40:25 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.329533412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration:
2.282454302s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:39:53 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.282498528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................]
duration:
2.381653551s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:38:54 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.38167654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................]
duration:
2.277196943s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:38:01 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.277290532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................]
duration:
2.613210931s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:37:02 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.613256844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................]
duration:
2.526564214s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:36:30 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.526593592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................]
duration:
2.340447782s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:35:23 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.3405602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................]
duration:
2.47547891s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:34:53 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.475512061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................]
duration:
2.485105447s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:34:05 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.485136061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................]
duration:
2.300538288s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:33:23 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.30056779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................]
duration:
2.329914439s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:32:39 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.329943479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................]
duration:
2.305499789s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:31:55 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.305522301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................]
duration:
2.479818228s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:30:46 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.479882282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................]
duration:
2.385021049s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:30:24 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.385045516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................]
duration:
2.475719919s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:29:39 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.475746251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................]
duration:
2.373894103s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:29:05 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.373922724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................]
duration:
2.390345915s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:27:54 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.390372897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................]
duration:
2.312163907s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:27:21 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.312205946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................]
duration:
2.395550729s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:26:48 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.395565771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............]
duration:
2.353694155s
trace:
https://jaeger.dev.storj.io/trace/1751ea226e84ac5d
FAIL (2025-12-14 14:26:15 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload
Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload
Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments...
Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false
Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment...
Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment.
Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece...
Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589     , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841     , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment
Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments...
Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265   , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979   , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158   , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230   , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302   , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411   , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655   , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337   , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409   , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877     , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520     , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835   , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil>
Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590     , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000
Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734     , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF
Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments...
Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.35372269s core uplink us-select-1