Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.341424907s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:54:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.34145717s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 165.464735ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:54:15 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 165.489656ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.440098041s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:54:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.440149118s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.438828441s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:53:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.438889548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.380098704s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:53:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.380124325s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 163.47303ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:53:17 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 163.498209ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.572742634s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.572767404s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 186.848048ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:52:20 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 186.878222ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.432289735s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:51:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.432315648s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 157.419569ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:51:28 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 157.445602ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.388926207s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388952461s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 165.165812ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:50:36 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 165.191352ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.259857847s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:50:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.259885615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.324469387s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:49:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.324498418s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 163.901795ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:49:50 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 163.993605ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 170.604291ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:49:30 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 170.630994ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.28359372s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:49:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.283620814s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 155.767401ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:48:33 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 155.792771ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.319924136s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:48:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.319952327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.516328765s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:47:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.516358868s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 184.035129ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:47:19 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 184.081406ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 99.550284ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Content-Type=text/plain; charset=UTF-8 Content-Length=15 Cache-Control=private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires=Thu, 01 Jan 1970 00:00:01 GMT Cf-Ray=9ade7e5d7aae9963-FRA X-Frame-Options=SAMEORIGIN Date=Sun, 14 Dec 2025 14:46:56 GMT Referrer-Policy=same-origin Proxy-Status=Cloudflare-Proxy;error=http_response_incomplete target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/1244b8592b06e8c8 FAIL (2025-12-14 14:46:56 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "error code: 500" Duration: 99.577018ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.4382214s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:46:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.43832894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.420817711s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:46:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.420848188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.3816382s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:45:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381668039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.346881105s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:44:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.346910755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.331783963s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.331816542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.508647455s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.508678876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.484208804s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:41:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.484239362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.294311797s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:40:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.294340069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.329415289s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:40:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.329533412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.282454302s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.282498528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.381653551s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38167654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.277196943s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:38:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.277290532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.613210931s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.613256844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.526564214s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.526593592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.340447782s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:35:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.3405602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.47547891s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:34:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475512061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.485105447s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:34:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.485136061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.300538288s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:33:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.30056779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.329914439s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:32:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.329943479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.305499789s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.305522301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.479818228s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:30:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.479882282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.385021049s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.385045516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.475719919s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475746251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.373894103s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373922724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.390345915s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:27:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390372897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.312163907s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.312205946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.395550729s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:26:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395565771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.353694155s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.35372269s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)