Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.3816382s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:45:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381668039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.346881105s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:44:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.346910755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.331783963s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.331816542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.508647455s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.508678876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.484208804s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:41:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.484239362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.294311797s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:40:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.294340069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.329415289s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:40:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.329533412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.282454302s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:39:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.282498528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.381653551s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38167654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.277196943s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:38:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.277290532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.613210931s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.613256844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.526564214s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.526593592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.340447782s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:35:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.3405602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.47547891s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:34:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475512061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.485105447s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:34:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.485136061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.300538288s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:33:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.30056779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.329914439s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:32:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.329943479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.305499789s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.305522301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.479818228s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:30:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.479882282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.385021049s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.385045516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.475719919s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.475746251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.373894103s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373922724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.390345915s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:27:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390372897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.312163907s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.312205946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.395550729s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:26:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.395565771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.353694155s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:26:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.35372269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.454782164s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:25:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.45480946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.38847764s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:25:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388504139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.275965779s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:25:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.275990611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.253782602s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:24:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.253812123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.3256927s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:23:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.325721529s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.221129858s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:22:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.22117078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.473623937s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:21:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.473913182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.295058856s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:20:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.295088694s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.367238058s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:19:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.367263244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.31474311s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:19:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.314769275s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.546232843s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:18:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.546260553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.346671838s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:18:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.346698501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.306099399s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:17:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.306130685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.322354294s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:16:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.322379442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.114496676s trace: https://jaeger.dev.storj.io/trace/1751ea226e84ac5d FAIL (2025-12-14 14:16:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 14:16:03.322622529 (214h47m43.302975166s) [upload=188771]: Starting upload Dec 14 14:16:03.322698383 (214h47m43.303051027s) [upload=188771]: Done starting upload Dec 14 14:16:03.322957650 (214h47m43.303310288s) [upload=188771]: Uploading segments... Dec 14 14:16:03.323047310 (214h47m43.303399971s) [upload=188771]: Got next segment. Inline: false Dec 14 14:16:03.323095962 (214h47m43.30344861s) [upload=188771]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 14:16:03.323104243 (214h47m43.303456893s) [upload=188771]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 14:16:03.446699051 (214h47m43.427051694s) [seg_pos={0 0}, upload=188771]: Begin upload segment... Dec 14 14:16:03.446762560 (214h47m43.427115199s) [seg_pos={0 0}, upload=188771]: Done begin upload segment. Dec 14 14:16:03.446782966 (214h47m43.427135626s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446798773 (214h47m43.427151422s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.446883480 (214h47m43.427236128s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.447750216 (214h47m43.428102851s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448003133 (214h47m43.428355855s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.448960450 (214h47m43.429313088s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.450289219 (214h47m43.43064186s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451155877 (214h47m43.431508525s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.451700169 (214h47m43.432052812s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453101406 (214h47m43.433454055s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453306320 (214h47m43.433658971s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.453902924 (214h47m43.43425557s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.454485471 (214h47m43.434838121s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455040478 (214h47m43.435393125s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455598935 (214h47m43.435951585s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.455867680 (214h47m43.436220329s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Uploading piece... Dec 14 14:16:03.861047945 (214h47m43.841400591s) [noise=true , address=77.90.31.12:4589 , piece_id=H4KRSS3JIDMWH3XPAECKFWD5TLV667ID2ZSZ7SQ4JHRV2RS3E2ZQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:03.871023383 (214h47m43.851376028s) [noise=true , address=77.90.31.19:4841 , piece_id=R2PNSFVNSXCIKEKQNTIUOKZ5OPHPK6NU6EE3K54FAGOIG2NE6ENQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:04.080925197 (214h47m44.061277846s) [upload=188771]: Next returned nil segment Dec 14 14:16:04.080944439 (214h47m44.061297107s) [upload=188771]: Waiting for error group managing segments... Dec 14 14:16:04.209819284 (214h47m44.190171943s) [noise=true , address=145.79.227.47:4265 , piece_id=3QHWAYPXPG34D2QAFXMRIXLUBAX3SVTVCLYR3DTJ6WRWEPOU3PKQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.233005644 (214h47m44.213358285s) [noise=true , address=145.79.227.38:4979 , piece_id=ZZXBSNY2YNSNNXTBGCVI2J27VG4DYOWZWHCB3BZG6KCQXWGEZNFQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.234227859 (214h47m44.214580501s) [noise=true , address=145.79.227.43:4158 , piece_id=RLAN2IHF643NXKXXREGGR5BFI5GYDBCD6UF27YJRFP5M7HPUQ4YQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239681369 (214h47m44.22003402s) [noise=true , address=145.79.227.45:4230 , piece_id=HKMA7J4GJPBAS2L3W64H7WG7H3Z2JZTPBN3E5PI4L4WSGSAEIGFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.239894423 (214h47m44.220247063s) [noise=true , address=145.79.227.48:4302 , piece_id=3YPBISDZHUXSROSFRMCJHQDPPCQ3CG3IKQRBXLHOZIYVDOGC7BTQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.240454771 (214h47m44.220807417s) [noise=true , address=145.79.227.51:4411 , piece_id=VCGINQEQMBEB4PUMBQOXB62UZJ7AXEN7SUSJOS45RGTV4SBIKUYQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.241497930 (214h47m44.221850571s) [noise=true , address=145.79.227.26:4655 , piece_id=3T2RNV6KR3LUYI4AY2HJ2ABHQTELURMGS6WSGZW6UQQUDPZW2GQQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.242469256 (214h47m44.222821901s) [noise=true , address=145.79.227.49:4337 , piece_id=JAA6NLOVKMU2JNEE2GENAQMRODTXVEP6FI3ZXNV63FY7LF3RYYFA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.243873927 (214h47m44.224226573s) [noise=true , address=145.79.227.51:4409 , piece_id=JPCGGAZ7GOZ6XBE4RDYV6W6JBDCYHDQ5SWUH3C5N5QC74I2E2OXQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.244902987 (214h47m44.225255626s) [noise=true , address=77.90.31.21:4877 , piece_id=DLV3ZFIPCHKY3VC5HCPPYIFY527EGI7VUXGM5URKQRLI35QFDBLA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.246138337 (214h47m44.226490985s) [noise=true , address=77.90.31.10:4520 , piece_id=UGV5ITABBYJWVGJS3MN2VUU7IAGX4L5D6NLIHSVZANWUMGGEV2OQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.284989325 (214h47m44.265341963s) [noise=true , address=145.79.227.32:4835 , piece_id=AAS74W2QEXQHBTQSLASQZQSDRKB25ONCUJU4Q44J3KTSIIKGUTDQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: <nil> Dec 14 14:16:04.787553216 (214h47m44.767905858s) [noise=true , address=77.90.31.12:4590 , piece_id=X55W4DSDFO6F4R3YS5EKB3DH4FCBZOFLFC2IH2SDD6AMKPXDSZIQ, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: storage node overloaded, request limit: 2000 Dec 14 14:16:33.026981756 (214h48m13.007334401s) [noise=true , address=77.90.31.16:4734 , piece_id=KQCIKRMDXIY5LUPJVINOLAU6TVURI5Z5MOFC7IDRZKILR2JIN2OA, seg_pos={0 0}, upload=188771]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 14:16:33.027035288 (214h48m13.007387915s) [seg_pos={0 0}, upload=188771]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 14:16:33.139454175 (214h48m13.119806811s) [upload=188771]: Done uploading segments... Dec 14 14:16:33.139495796 (214h48m13.119848423s) [upload=188771]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.114520169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.481232176s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:28:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.481259773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.77706044s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:27:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.777091051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.701472387s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:26:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.701498286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 3.783147394s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:25:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.783172497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.756668318s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:24:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.756711658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 3.662425383s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:23:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.662451282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 4.305857665s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:22:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.30589618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 4.175762084s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:21:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.175788617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.335462051s trace: https://jaeger.dev.storj.io/trace/559421ed0d7f742a FAIL (2025-12-14 13:20:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 14 12:41:06.626928803 (213h12m46.607281425s) [upload=187381]: Starting upload Dec 14 12:41:06.626995239 (213h12m46.607347876s) [upload=187381]: Done starting upload Dec 14 12:41:06.627174396 (213h12m46.607527035s) [upload=187381]: Uploading segments... Dec 14 12:41:06.627394315 (213h12m46.607746958s) [upload=187381]: Got next segment. Inline: false Dec 14 12:41:06.627450481 (213h12m46.607803129s) [upload=187381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 14 12:41:06.627478972 (213h12m46.607831618s) [upload=187381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 14 12:41:06.740140944 (213h12m46.720493581s) [seg_pos={0 0}, upload=187381]: Begin upload segment... Dec 14 12:41:06.740215601 (213h12m46.720568238s) [seg_pos={0 0}, upload=187381]: Done begin upload segment. Dec 14 12:41:06.740348817 (213h12m46.720701462s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.740403732 (213h12m46.720756379s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.741125276 (213h12m46.721477922s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742126247 (213h12m46.72247889s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.742708229 (213h12m46.723060871s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743268942 (213h12m46.723621591s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.743949949 (213h12m46.724302591s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.744522590 (213h12m46.724875232s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.745178970 (213h12m46.725531612s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.747159347 (213h12m46.727511972s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.750177640 (213h12m46.730530284s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752279475 (213h12m46.732632117s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.752731748 (213h12m46.73308439s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754280053 (213h12m46.734632698s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.754523018 (213h12m46.734875665s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:06.755952837 (213h12m46.736305473s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Uploading piece... Dec 14 12:41:07.136429866 (213h12m47.116782511s) [noise=true , address=77.90.31.19:4842 , piece_id=BZCJIQKRRDVJKNEAW3SBDB2DFCZ7GBIUJJMZMMF5XYXW6MN6OM7A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.139430152 (213h12m47.119782795s) [noise=true , address=77.90.31.12:4589 , piece_id=6I7R4742WGI5K5XBS6V4QQB4VR4YXKPZJKFQGKCLGUV4XPB4SFIQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:07.347716267 (213h12m47.328068916s) [upload=187381]: Next returned nil segment Dec 14 12:41:07.347737269 (213h12m47.328089919s) [upload=187381]: Waiting for error group managing segments... Dec 14 12:41:07.476661116 (213h12m47.457013753s) [noise=true , address=77.90.31.15:4697 , piece_id=V7Q5ERS63JWZJL2LSL2LE24DII6BQPJMMDDDV7L7DALRY3JZHQDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.487402813 (213h12m47.467755462s) [noise=true , address=145.79.227.31:4797 , piece_id=UUIWN5RO6APYMMOB22BFD73ZD3RTND55SQ3TPJNR5Y4TNGP3N5EQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.488415319 (213h12m47.468767963s) [noise=true , address=145.79.227.51:4411 , piece_id=RSOU2GML574VWCMIWT47TMRGRSQADNRMRGGJB4LFKDKIK7IVAHZA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.489593720 (213h12m47.469946362s) [noise=true , address=145.79.227.43:4157 , piece_id=XUWYQBVQMVAXVX2CD2D24RDD7JEDZL7KXHHY4UGWNND4GT6UVBRA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.495500548 (213h12m47.47585319s) [noise=true , address=145.79.227.52:4446 , piece_id=DNO2WOYSRRKELG3B4NFDG6VUY5472RCZVAQ5CAZDI5NX5QNN247A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.497152594 (213h12m47.477505235s) [noise=true , address=145.79.227.45:4230 , piece_id=TIOSSQS3YKVAFLTPHPAF42DQXCWL43DVIHMDSXMHTOGWH44MOFHQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510366465 (213h12m47.490719107s) [noise=true , address=145.79.227.49:4338 , piece_id=4Q5OCSEI4Y76XJSNP73ATCCUXI5DH4ATBCKG2USPKQCRD46G7WKA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.510963788 (213h12m47.491316425s) [noise=true , address=145.79.227.37:4565 , piece_id=VT7WWQVIK62PEXXFMYHQNXDA7EICL5U3A3QFAWIXYKMANPJ5AGKQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.514230398 (213h12m47.494593664s) [noise=true , address=145.79.227.47:4267 , piece_id=VKCVNRYOLT5V6R6ZSQFOOL5Z5QTDT3R7GSFWONSJAYWBWZ3K3BWA, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.518472285 (213h12m47.498824927s) [noise=true , address=145.79.227.39:4015 , piece_id=AJGVTTJEAMXJKYW2DRXHOJBYGEECQUFZD44IYQQHS4LYPDXCSEDQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.521580994 (213h12m47.501933636s) [noise=true , address=145.79.227.33:4870 , piece_id=4XW7PYFBZGV5UBV43U4PTQCIIDLMCOO7RHW4VBWMOVWGJQWMUS6A, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:07.822126868 (213h12m47.802479508s) [noise=true , address=145.79.227.30:4763 , piece_id=QTYJHY3SQZGOOKXA7UCOFHTIIH6U7I64RCMN3VLHLFR3JTTB63MQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: storage node overloaded, request limit: 2000 Dec 14 12:41:11.312666958 (213h12m51.293019599s) [noise=true , address=145.79.227.30:4762 , piece_id=A2CVH2B5SG34T5JGFNCGB6MMY7IHZZMWBYGOSAZEHITPP6OVDMWQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: <nil> Dec 14 12:41:36.331935751 (213h13m16.312288397s) [noise=true , address=77.90.31.16:4735 , piece_id=VLADFRUYV6YSX4RXICDCYD7EY6C5AJXHZ3M7GVYS7VDJZMUESSRQ, seg_pos={0 0}, upload=187381]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 14 12:41:36.332031918 (213h13m16.312384557s) [seg_pos={0 0}, upload=187381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 14 12:41:36.447821678 (213h13m16.428174314s) [upload=187381]: Done uploading segments... Dec 14 12:41:36.447860513 (213h13m16.428213157s) [upload=187381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.335489764s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)