Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.516397962s FAIL (2025-07-18 02:55:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.516421155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.82507734s FAIL (2025-07-18 02:54:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.825165311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.509510691s FAIL (2025-07-18 02:53:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.50954608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.565751647s FAIL (2025-07-18 02:52:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.565781277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.656589827s FAIL (2025-07-18 02:51:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.656624103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 2.710503882s FAIL (2025-07-18 02:51:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.710530044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.561374617s FAIL (2025-07-18 02:50:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.56140381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.35929279s FAIL (2025-07-18 02:49:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.359326118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 2.416186237s FAIL (2025-07-18 02:48:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.416231843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.639854629s FAIL (2025-07-18 02:48:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.639902256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 2.324934397s FAIL (2025-07-18 02:47:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.324967612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.345483796s FAIL (2025-07-18 02:46:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.345519793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 2.665397611s FAIL (2025-07-18 02:46:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.665429456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 2.564739555s FAIL (2025-07-18 02:45:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.564768483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 2.538131635s FAIL (2025-07-18 02:45:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.538253494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 2.570901689s FAIL (2025-07-18 02:44:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.570940195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.708406176s FAIL (2025-07-18 02:43:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.708440183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.562111168s FAIL (2025-07-18 02:42:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.562143533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 2.945093417s FAIL (2025-07-18 02:41:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.945127773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 2.613228137s FAIL (2025-07-18 02:40:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.613260721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 2.478151549s FAIL (2025-07-18 02:39:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.478178712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 2.455627295s FAIL (2025-07-18 02:38:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.455673909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 2.596153247s FAIL (2025-07-18 02:38:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.596178193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 2.651819818s FAIL (2025-07-18 02:37:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.651890801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 2.485964793s FAIL (2025-07-18 02:36:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.485997464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 2.586892636s FAIL (2025-07-18 02:35:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.586917104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 2.538999716s FAIL (2025-07-18 02:35:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.539086437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 2.680417919s FAIL (2025-07-18 02:34:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.680452699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 2.382055776s FAIL (2025-07-18 02:33:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.382086807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 2.545015065s FAIL (2025-07-18 02:33:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.545052108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 2.589035672s FAIL (2025-07-18 02:32:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.589073676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 2.422620453s FAIL (2025-07-18 02:31:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.422662822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 2.528182832s FAIL (2025-07-18 02:29:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.528209874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 2.45134459s FAIL (2025-07-18 02:29:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.451376842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.570157046s FAIL (2025-07-18 02:28:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.570191018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.507833364s FAIL (2025-07-18 02:27:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.50786569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.654711069s FAIL (2025-07-18 02:26:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.654740872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 2.629313665s FAIL (2025-07-18 02:26:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.629344352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 2.768443759s FAIL (2025-07-18 02:25:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.768476809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.684489499s FAIL (2025-07-18 02:24:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.684528217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 2.504792778s FAIL (2025-07-18 02:23:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.504822651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 2.508420913s FAIL (2025-07-18 02:23:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.508448879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 2.695745268s FAIL (2025-07-18 02:22:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.695869325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 2.771041065s FAIL (2025-07-18 02:21:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.771085832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 2.627154905s FAIL (2025-07-18 02:20:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.627178075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 2.930494653s FAIL (2025-07-18 02:19:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.930566162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 2.643211978s FAIL (2025-07-18 02:18:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.643247152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 2.726656922s FAIL (2025-07-18 02:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.726700555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 2.605347549s FAIL (2025-07-18 02:17:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.60538165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.630222197s FAIL (2025-07-18 02:16:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 0468 (593h2m18.628194085s) [upload=474167]: Got next segment. Inline: false Jul 18 02:13:18.744827419 (593h2m18.628241045s) [upload=474167]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 02:13:18.744833270 (593h2m18.628246896s) [upload=474167]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 02:13:18.944247732 (593h2m18.827661359s) [seg_pos={0 0}, upload=474167]: Begin upload segment... Jul 18 02:13:18.944300780 (593h2m18.827714397s) [seg_pos={0 0}, upload=474167]: Done begin upload segment. Jul 18 02:13:18.944327501 (593h2m18.827741131s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.944748561 (593h2m18.82816219s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945029770 (593h2m18.828443399s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945305573 (593h2m18.828719199s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945587945 (593h2m18.829001572s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.945905736 (593h2m18.829319373s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946318615 (593h2m18.829732242s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946635972 (593h2m18.830049598s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.946932847 (593h2m18.830346512s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947207379 (593h2m18.830621008s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947511937 (593h2m18.830925567s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.947896979 (593h2m18.831310619s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948220274 (593h2m18.831633901s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948536793 (593h2m18.831950434s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.948900904 (593h2m18.832314537s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949201483 (593h2m18.832615112s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949472498 (593h2m18.832886128s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:18.949792417 (593h2m18.833206048s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Uploading piece... Jul 18 02:13:19.506882340 (593h2m19.390295955s) [upload=474167]: Next returned nil segment Jul 18 02:13:19.506923123 (593h2m19.390336751s) [upload=474167]: Waiting for error group managing segments... Jul 18 02:13:19.666828456 (593h2m19.550242074s) [noise=true , address=45.91.116.5:4315 , piece_id=7B4KSGICA5N2MM2TO52UHHKYUOXS2AQHYORMC2TC6RCYAODHV4HQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.669997440 (593h2m19.55341106s) [noise=true , address=45.91.116.8:4075 , piece_id=R63TYEY34PGAQB3QZ3EKCVBZJ6PY3XRZ7FZMPADWGLAWSRR3BV7A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.678533346 (593h2m19.561946983s) [noise=true , address=45.91.116.7:4013 , piece_id=IOSFQDAAUSXBLF7KJFK36SAPUFTIOUOGD65K3KNRUCKMG2NV5M6A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688823710 (593h2m19.57223733s) [noise=true , address=79.127.219.51:4197 , piece_id=C5PCVWGRXFLMSKY3JIQ34XAWC5CY4JR3XPPSGQ5POIFTA2AVM52Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.688899040 (593h2m19.572312661s) [noise=true , address=45.91.116.12:4236 , piece_id=LC7EETGY3NJHRVPE2TFBJWPGFCAMOFV47R2A3TQEIWJ5N6V4264A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.692235863 (593h2m19.575649484s) [noise=true , address=45.91.116.11:4122 , piece_id=2N2PKPMW7MMQKEGBJ5JALY6GA6PQ7H2Z2RV6BRSOTOLLDWGGWEWA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.696462183 (593h2m19.579875811s) [noise=true , address=104.36.82.40:4965 , piece_id=QVBRNABGNUDALSD6YSERJIGPRAVMRCBNZWD6S3L7UE7UMDINXIZQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.813900946 (593h2m19.697314563s) [noise=true , address=45.91.95.4:4023 , piece_id=MHI6MJN3IONG7D6VL5JDWW7J7TZQ5P55TXV6RX6MZZ7M5DYWEOTQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:19.821077860 (593h2m19.70449148s) [noise=true , address=45.91.95.2:4027 , piece_id=O6OHEXXYEZVJYIQSQRAJ4P225SMCRRERKFZD5SPPSJRLSSH7UFMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.204635402 (593h2m20.088049022s) [noise=true , address=64.49.13.140:4514 , piece_id=AHOWVODFU6B3AAANVVFU4QZDTRXYMEXWAPHMIUBL445N4QBNBDPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.284751010 (593h2m20.168164626s) [noise=true , address=79.127.205.233:4655 , piece_id=K7FWZFKAEWHXTY5QOHMRUNIS4VBHG6OOVLXBF7ZYWV5LRYSGMVXA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:20.285897562 (593h2m20.169311174s) [noise=true , address=79.127.205.234:4617 , piece_id=OXKUU4PF5ZSYD2LTNHJ367GJXCQXDRNGUY2RH3UI7ANAZKNGHE3Q, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:13:22.157138533 (593h2m22.040552152s) [noise=true , address=104.36.82.5:4866 , piece_id=DN7REXMIHHAYHR3YV4YZ7RIPVWMR42AX6HL46XILWSWHQTXDN2WA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: <nil> Jul 18 02:14:18.469416431 (593h3m18.352830058s) [noise=true , address=77.90.31.15:4707 , piece_id=HEGKDFTKYYIKXRMDOKXOJRTSMAUUBL2D7UBCINKSKQV5KNNRNW3A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469487502 (593h3m18.352901118s) [noise=true , address=77.90.31.10:4546 , piece_id=2VKHQSWRL5HYCWMY6463GSMQDOV3GF2GQQF5DKWTA3QSVKG3G2QQ, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK, address:77.90.31.10:4546): protocol: write tcp 139.59.144.22:58640->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469495352 (593h3m18.352908971s) [noise=true , address=77.90.31.18:4817 , piece_id=7FZU2HQAWBWMSZ2ZHKLPJRPUKQHNSIXF3O2DXNIC5E3BNKZQ7MMA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469551424 (593h3m18.35296505s) [noise=true , address=77.90.31.13:4645 , piece_id=6AGW7BEMYHIL7BOAUETZWEHJM36HUBJ2HYPNXYPG3FHC2KJYKS2A, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: write tcp 139.59.144.22:33516->77.90.31.13:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 02:14:18.469590566 (593h3m18.353004192s) [noise=true , address=77.90.31.11:4575 , piece_id=3YZHHBMW2TUSEH5GX7BQF65PXL7ZVUA646JRNBFF3NTCS7BCEQPA, seg_pos={0 0}, upload=474167]: Done uploading piece. err: ecclient: upload failed (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc, address:77.90.31.11:4575): protocol: write tcp 139.59.144.22:46706->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 18 02:14:18.469617916 (593h3m18.353031533s) [seg_pos={0 0}, upload=474167]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 18 02:14:18.582376941 (593h3m18.46579056s) [upload=474167]: Done uploading segments... Jul 18 02:14:18.582410612 (593h3m18.465824233s) [upload=474167]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.630255884s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)