Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.355805497s FAIL (2025-07-18 13:25:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.355837779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.410600204s FAIL (2025-07-18 13:25:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.410629328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.486000236s FAIL (2025-07-18 13:24:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.48602658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.523074096s FAIL (2025-07-18 13:23:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.523104446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.326149075s FAIL (2025-07-18 13:23:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326220789s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.275888047s FAIL (2025-07-18 13:22:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.275982733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.407600179s FAIL (2025-07-18 13:21:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.407627309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.284151426s FAIL (2025-07-18 13:20:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.284184841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.460406556s FAIL (2025-07-18 13:19:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.460431044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.351177845s FAIL (2025-07-18 13:18:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.351211903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.321501379s FAIL (2025-07-18 13:17:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.321532894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.335982547s FAIL (2025-07-18 13:16:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.336058861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.301575252s FAIL (2025-07-18 13:15:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.301602453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.359782622s FAIL (2025-07-18 13:15:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359809899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.341578995s FAIL (2025-07-18 13:15:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.341612396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.478705076s FAIL (2025-07-18 13:14:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.478746639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.393288614s FAIL (2025-07-18 13:13:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.393330323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.402658934s FAIL (2025-07-18 13:12:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402687507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.402361377s FAIL (2025-07-18 13:11:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402391338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.500465529s FAIL (2025-07-18 13:10:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.500492246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.374894584s FAIL (2025-07-18 13:09:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374930934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.286646977s FAIL (2025-07-18 13:08:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.286676707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.323180596s FAIL (2025-07-18 13:07:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.323214999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.354887627s FAIL (2025-07-18 13:07:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.354924448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.337468267s FAIL (2025-07-18 13:05:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33750139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.36533549s FAIL (2025-07-18 13:05:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.365366256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.350278346s FAIL (2025-07-18 13:04:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.350304009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.390864496s FAIL (2025-07-18 13:03:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.390907771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.409249365s FAIL (2025-07-18 13:02:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.409282063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.372068913s FAIL (2025-07-18 13:02:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.372089902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.384232867s FAIL (2025-07-18 13:01:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.384255727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.312312993s FAIL (2025-07-18 13:01:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.312346744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.125348033s FAIL (2025-07-18 13:01:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== address=104.36.82.8:4972 , piece_id=77RPGWWVXQR4LQEIW762C2RFWHGJJW6EXIWOMXAGQDY45MC4VWPQ, seg_pos={0 0}, upload=482770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Jul 18 12:59:31.626841382 (603h48m31.510254996s) [seg_pos={0 0}, upload=482770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:59:31.626960601 (603h48m31.51037423s) [upload=482770]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:59:31.626969171 (603h48m31.510382799s) [upload=482770]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:59:31.790603737 (603h48m31.674017374s) [upload=482770]: Done uploading segments... Jul 18 12:59:31.790628230 (603h48m31.674041864s) [upload=482770]: Upload finished. err: <nil> Jul 18 13:00:35.074453482 (603h49m34.957867106s) [upload=482783]: Starting upload Jul 18 13:00:35.074556815 (603h49m34.957970442s) [upload=482783]: Done starting upload Jul 18 13:00:35.074635767 (603h49m34.958049394s) [upload=482783]: Uploading segments... Jul 18 13:00:35.074876645 (603h49m34.958290261s) [upload=482783]: Got next segment. Inline: false Jul 18 13:00:35.074949993 (603h49m34.958363619s) [upload=482783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 13:00:35.074956646 (603h49m34.958370272s) [upload=482783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 13:00:35.193185275 (603h49m35.076598891s) [seg_pos={0 0}, upload=482783]: Begin upload segment... Jul 18 13:00:35.193234456 (603h49m35.076648076s) [seg_pos={0 0}, upload=482783]: Done begin upload segment. Jul 18 13:00:35.193260018 (603h49m35.076673647s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193277223 (603h49m35.076690861s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.193766117 (603h49m35.077179752s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194129526 (603h49m35.077543168s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194253230 (603h49m35.077666861s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194605148 (603h49m35.078018844s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.194841758 (603h49m35.078255395s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195038513 (603h49m35.078452139s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195448913 (603h49m35.078862542s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195565928 (603h49m35.078979559s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.195800132 (603h49m35.079213759s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196204612 (603h49m35.079618241s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.196591733 (603h49m35.080005361s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.199151660 (603h49m35.082565305s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201199167 (603h49m35.084612803s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.201675495 (603h49m35.085089124s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Uploading piece... Jul 18 13:00:35.803760706 (603h49m35.687174322s) [upload=482783]: Next returned nil segment Jul 18 13:00:35.803782913 (603h49m35.687196557s) [upload=482783]: Waiting for error group managing segments... Jul 18 13:00:35.938771156 (603h49m35.822184775s) [noise=true , address=77.90.31.10:4518 , piece_id=4YHAPEZ6NMEMRB3J6UL2LGOT5H4VP4NE5KBY2F4UGLVED6SLNAGA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.941693141 (603h49m35.825106766s) [noise=true , address=77.90.31.17:4770 , piece_id=LJN66JRLWJIGGDXAFCLAPBNG46T6DEDC4V2WRDTANDMXPSJKET2A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.943147300 (603h49m35.826560916s) [noise=true , address=45.91.116.13:4153 , piece_id=O3SDAQ5PAGGT6MNKVSDJMLV3IJMU5KSGT5CRF2PO5PHD56DZUQCQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.947243812 (603h49m35.830657439s) [noise=true , address=45.91.116.14:4190 , piece_id=M4VHAZH37NMKKBVUZEDV5PPXGZNBRP5N6GVXDBBRXFJGTYGK3AJA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.958584341 (603h49m35.841997969s) [noise=true , address=77.90.31.21:4879 , piece_id=AP2VRDVYLQ3F6XVBLLHS7LBG457RN2C7NRSRDC26BGAPM7XBY6YQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.960888979 (603h49m35.844302598s) [noise=true , address=45.91.116.14:4189 , piece_id=PG4HWV5FE4ONWQUGPSHQIUUN6F4VFF4B5LAKU6JQBXGOLQQZOVGQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.961358801 (603h49m35.844772419s) [noise=true , address=45.91.116.9:4263 , piece_id=FBL2TGPDGOLGNXEJ5FQROB4TL4BKQL4VV3HAIGUHTRM745CDQPFA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.965896672 (603h49m35.849310291s) [noise=true , address=45.91.116.6:4974 , piece_id=56N2YYAP6VCK7OAU43AMXNFGQICXAITF4WT3XDI7EJIH72UBSHLA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.975778905 (603h49m35.859192534s) [noise=true , address=104.36.82.4:4804 , piece_id=MORFWMHVIQ2N4A4GL5GJN4XXGHS2SOSQUSMTTF2B7H6GZVTGUZHQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.982203710 (603h49m35.865617342s) [noise=true , address=104.36.82.15:4259 , piece_id=UOZ6VDRTZBOJTXNJG2FWGWVMXGZLACNAKIJW2B6RXIRWKSCNDYXA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.983091582 (603h49m35.866505201s) [noise=true , address=104.36.82.6:4876 , piece_id=ASZGUMMFBYVOVJX724CO3VY2IYHR3JSPESLTIBVCRQ4KOWZEWZQA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:00:35.992479163 (603h49m35.87589279s) [noise=true , address=104.36.82.8:4971 , piece_id=DZXUREKPIAQHSNZSSQDU5PWG6PIKOGNLLZE2VNASZNBCHWGILE4A, seg_pos={0 0}, upload=482783]: Done uploading piece. err: <nil> Jul 18 13:01:04.802196041 (603h50m4.685609667s) [noise=true , address=77.90.31.17:4771 , piece_id=FO2NTICCWNHHVD2TRWN5GZWVHLJKA7A6YRUYKLWQKNARVZCYM5SA, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802245224 (603h50m4.685658866s) [noise=true , address=77.90.31.13:4627 , piece_id=Q4ORZY5VGWU6WIF5JXAZY2IYLAPZHVKKASBFPH3ZXTWDK2IIYIAQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: context deadline exceeded; context deadline exceeded Jul 18 13:01:04.802281412 (603h50m4.685695037s) [noise=true , address=77.90.31.16:4734 , piece_id=BQ3VJK4MFSPK5QTQWBZJNT2GDGX73BWLL4YINHZL6XIPCJIFQENQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802333980 (603h50m4.685747608s) [noise=true , address=77.90.31.16:4735 , piece_id=QYWEBBAWIVECIAZDSEDOZONV7GIL3OKENATD3W6GTKB2F2SW47OQ, seg_pos={0 0}, upload=482783]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 139.59.144.22:48834->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 13:01:04.802362907 (603h50m4.685776539s) [seg_pos={0 0}, upload=482783]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 18 13:01:04.924872463 (603h50m4.808286102s) [upload=482783]: Done uploading segments... Jul 18 13:01:04.924943255 (603h50m4.808356894s) [upload=482783]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.125399326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.531592342s FAIL (2025-07-18 12:55:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.531625346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.686647004s FAIL (2025-07-18 12:55:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.686676558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.743524547s FAIL (2025-07-18 12:54:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.743550403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.377668241s FAIL (2025-07-18 12:54:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.377699093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.389332118s FAIL (2025-07-18 12:53:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.389365351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.209215483s FAIL (2025-07-18 12:53:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.209305957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.340447604s FAIL (2025-07-18 12:53:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.340481381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.631982201s FAIL (2025-07-18 12:53:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.632012679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.562951054s FAIL (2025-07-18 12:52:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.562982651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.35639964s FAIL (2025-07-18 12:52:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.356438917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.607750572s FAIL (2025-07-18 12:52:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.607775461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.374164017s FAIL (2025-07-18 12:52:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.374197447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.459739562s FAIL (2025-07-18 12:51:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.459773151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.634880056s FAIL (2025-07-18 12:51:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.634923613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.418689584s FAIL (2025-07-18 12:51:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.418801826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.591654723s FAIL (2025-07-18 12:50:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== estItem_SegmentBegin Jul 18 12:13:33.620392272 (603h2m33.503805888s) [seg_pos={0 0}, upload=482147]: Begin upload segment... Jul 18 12:13:33.620453056 (603h2m33.503866676s) [seg_pos={0 0}, upload=482147]: Done begin upload segment. Jul 18 12:13:33.620494932 (603h2m33.503908562s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620500819 (603h2m33.503914451s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620905481 (603h2m33.504319113s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.620934724 (603h2m33.504348357s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621307624 (603h2m33.504721252s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621637637 (603h2m33.505051267s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.621886175 (603h2m33.505299802s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622021271 (603h2m33.505434904s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622324956 (603h2m33.505738588s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622569416 (603h2m33.505983029s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.622806676 (603h2m33.506220308s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623296793 (603h2m33.506710429s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623518504 (603h2m33.50693214s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.623790530 (603h2m33.507204156s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624033161 (603h2m33.507446793s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624144799 (603h2m33.507558431s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624501018 (603h2m33.507914648s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:33.624733995 (603h2m33.508147627s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Uploading piece... Jul 18 12:13:34.189792676 (603h2m34.073206312s) [upload=482147]: Next returned nil segment Jul 18 12:13:34.189817685 (603h2m34.073231318s) [upload=482147]: Waiting for error group managing segments... Jul 18 12:13:34.331211397 (603h2m34.214625009s) [noise=true , address=45.91.116.9:4283 , piece_id=AEYFP3WJB7T3V7KBY4CCZDCITOHRJ7KE3H7ZQHUPEACPWHB2KEYA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.344160118 (603h2m34.22757374s) [noise=true , address=45.91.116.14:4193 , piece_id=6EX7WNHS3GDEQC3RRDUPRY3DHYADVWKLUGPKBSRZJ2URA5Q2MTAA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.347105813 (603h2m34.23051943s) [noise=true , address=45.91.116.7:4031 , piece_id=H3PZRJYRVMM6JUDJBRRE7IF7WSF6B4XEHYHGY4S4J34YJOQPU7IQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.349058714 (603h2m34.232472333s) [noise=true , address=79.127.219.38:4023 , piece_id=3UQLEARQ5EQNY26WLIHBXTSCC44Q2PI76KEZLN3YQWE6PBR73LQQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.367227155 (603h2m34.250640774s) [noise=true , address=77.90.31.19:4868 , piece_id=TLFODQ6CPVWQKFXZMZ24HBW2KFSISPIWJGQVG4K4BLDNKVTTI6FA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.410633895 (603h2m34.294047529s) [noise=true , address=79.127.219.48:4082 , piece_id=5VAAIJBGYVT7YTVJWDMGQISS4OJEAPLZP43F3XAAUGZVUEALK7JA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.510862380 (603h2m34.394276009s) [noise=true , address=45.91.95.5:4013 , piece_id=5ACOTNNWUXY2LRDJEDKX43XFR2THEDDTVDOL6IEQGV4KEIJLWANQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.530632900 (603h2m34.41404653s) [noise=true , address=45.91.95.4:4004 , piece_id=P4QBLXZAIOWRG6AFQUDPQDF4IAZQP6W2DFKIIHULRJTBXKMJHGYQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.567043418 (603h2m34.45045705s) [noise=true , address=79.127.201.215:4344 , piece_id=SHRROJHGGJ3K43DOVGIILV5HDFKJESWFH5R4AEKXLJX7NWHLMFMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.903974872 (603h2m34.78738849s) [noise=true , address=79.127.205.245:4237 , piece_id=IHKGH3R7OXUGPWN5G24TQPRFMX7YUJ5555S4BMSTOSIHPSZVLZCA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:34.919166831 (603h2m34.802580447s) [noise=true , address=79.127.205.233:4628 , piece_id=ZZ2RAQP65MMFNU53UQKFXRR4QPMKJMEVM3GZAZDFOELWPMQ5PSWA, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:13:35.318955896 (603h2m35.202369514s) [noise=true , address=45.91.116.12:4245 , piece_id=GMKIQQKMH3VNH5TUFET2UNQZAV2FOLS22TNQYEWJSXLS5QOHJZ4Q, seg_pos={0 0}, upload=482147]: Done uploading piece. err: <nil> Jul 18 12:14:33.152770912 (603h3m33.03618453s) [noise=true , address=77.90.31.10:4544 , piece_id=HQ5NI3S2XMCQUD3ROOHDIZUN62P3242MPJGZEAWUSCK5ZAZCSPMQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1MwFhR5fkW2mNRpj7qAKGdRRC8ua2SBAppwWmyxYuhgpUkZETE, address:77.90.31.10:4544): protocol: write tcp 139.59.144.22:38070->77.90.31.10:4544: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152837337 (603h3m33.036250954s) [noise=true , address=77.90.31.15:4709 , piece_id=NVHHUBQDCG5YT2CLUI3DSOU5ALKVOE47TJUKQYJ5MNCFRAT6DRDQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: write tcp 139.59.144.22:35908->77.90.31.15:4709: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.152882004 (603h3m33.036295622s) [noise=true , address=77.90.31.13:4641 , piece_id=MNDISFBSOAOUTJSRNQC523M5CC4MLRVH4WAS3TAFRNUCYFJTSGHQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:187b9r6f2MK5c5mJEVasrhfpRHzhC4JbkWTbVvHuqB62XYs8xL, address:77.90.31.13:4641): protocol: write tcp 139.59.144.22:60742->77.90.31.13:4641: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153121796 (603h3m33.036535415s) [noise=true , address=77.90.31.16:4757 , piece_id=IKHQAHNZASFBPTVCK7HE3HREEAWP5GWPDZHBAU4EALF55MIN3GIQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, address:77.90.31.16:4757): protocol: write tcp 139.59.144.22:36646->77.90.31.16:4757: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153168438 (603h3m33.036582053s) [noise=true , address=77.90.31.12:4613 , piece_id=O6ZGBAXU2UV6HBIHZU4K5LRTMKKXC6GI63VTDYFHRRT46EX7Z53A, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:42646->77.90.31.12:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:14:33.153391026 (603h3m33.036804643s) [noise=true , address=77.90.31.11:4557 , piece_id=33SQ5A3RML2B77H4BDXU7SQ24WN2SXUEV64RGVXTCIZ6EBLLHDAQ, seg_pos={0 0}, upload=482147]: Done uploading piece. err: ecclient: upload failed (node:12jhwkR5WaB34mEb6JJP2Lnyvo4HdkghJeR3PGz1hTDs12oV6uX, address:77.90.31.11:4557): protocol: write tcp 139.59.144.22:48542->77.90.31.11:4557: use of closed network connection; context deadline exceeded Jul 18 12:14:33.153415039 (603h3m33.036828654s) [seg_pos={0 0}, upload=482147]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 18 12:14:33.271003419 (603h3m33.154417038s) [upload=482147]: Done uploading segments... Jul 18 12:14:33.271053119 (603h3m33.154466738s) [upload=482147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 2.591687352s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.414132315s FAIL (2025-07-18 12:50:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== twork connection; conn closed: op; piecestore: piecestore close: context canceled Jul 18 12:13:36.059334855 (603h2m35.942748487s) [seg_pos={0 0}, upload=482150]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 18 12:13:36.059429166 (603h2m35.942842897s) [upload=482150]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 18 12:13:36.059435794 (603h2m35.942849427s) [upload=482150]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 18 12:13:36.222045265 (603h2m36.105458897s) [upload=482150]: Done uploading segments... Jul 18 12:13:36.222068360 (603h2m36.105481987s) [upload=482150]: Upload finished. err: <nil> Jul 18 12:14:39.658268712 (603h3m39.541682338s) [upload=482161]: Starting upload Jul 18 12:14:39.658371478 (603h3m39.541785116s) [upload=482161]: Done starting upload Jul 18 12:14:39.658518044 (603h3m39.54193168s) [upload=482161]: Uploading segments... Jul 18 12:14:39.658896930 (603h3m39.542310547s) [upload=482161]: Got next segment. Inline: false Jul 18 12:14:39.658985463 (603h3m39.542399092s) [upload=482161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 18 12:14:39.658990950 (603h3m39.542404576s) [upload=482161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 18 12:14:39.770564164 (603h3m39.65397778s) [seg_pos={0 0}, upload=482161]: Begin upload segment... Jul 18 12:14:39.770628215 (603h3m39.654041844s) [seg_pos={0 0}, upload=482161]: Done begin upload segment. Jul 18 12:14:39.770660270 (603h3m39.654073903s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770686400 (603h3m39.654100038s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.770748767 (603h3m39.654162397s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771127939 (603h3m39.654541566s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771181512 (603h3m39.654595145s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.771620867 (603h3m39.655034504s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.774266589 (603h3m39.657680227s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776315497 (603h3m39.659729122s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776661976 (603h3m39.660075608s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.776932615 (603h3m39.660346241s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777195553 (603h3m39.66060918s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777496856 (603h3m39.660910481s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.777781639 (603h3m39.661195265s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778068328 (603h3m39.661481955s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778350883 (603h3m39.661764511s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:39.778821109 (603h3m39.662234744s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Uploading piece... Jul 18 12:14:40.395675287 (603h3m40.279088924s) [upload=482161]: Next returned nil segment Jul 18 12:14:40.395694996 (603h3m40.279108633s) [upload=482161]: Waiting for error group managing segments... Jul 18 12:14:40.525012265 (603h3m40.408425892s) [noise=true , address=45.91.116.9:4262 , piece_id=WD6Y5RRCTTKBNO7GB5ZUCRFKUSYXV7NIQZPZV3X4DGYYTS4JTZVQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.525513556 (603h3m40.408927173s) [noise=true , address=45.91.116.13:4153 , piece_id=7OW2QFFUPR4E2JT3EIWDFSPCNVYUSBC25TYYTCDCJBGCK5VMTMOQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.529347087 (603h3m40.412760705s) [noise=true , address=45.91.116.5:4297 , piece_id=IT7WPAMTGLRCXVTDVPXN4LNPBOHDBWFX7KL23E3TVJE2NKW74APQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.530235891 (603h3m40.413649507s) [noise=true , address=45.91.116.7:4009 , piece_id=6RNBTQCZ2BUOQ22NZ5U2IHTAP7PX5YVWXKLH24R2LLAHMENLXQ4Q, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.532741510 (603h3m40.416155123s) [noise=true , address=77.90.31.11:4554 , piece_id=R6SI3DQS4TBIDCUEONSKUAXIWH3RR2STUIZE5DVWSBNLNFAONHPQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.534588102 (603h3m40.41800172s) [noise=true , address=45.91.116.8:4047 , piece_id=X3GG25HH5X5CJCOSZUWFVCB4MCSEX56MCWX6XOFOV5JRQKID3TGA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.538981731 (603h3m40.422395355s) [noise=true , address=45.91.116.13:4154 , piece_id=DTOA6BCE6BKOKJWBPFG6VAKSDWUBT4BQOJMGKJZPVDKUB5B32JJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.543710393 (603h3m40.427124056s) [noise=true , address=45.91.116.14:4191 , piece_id=NB6T54P7GRT7N45BY35JKDUNIGRMFISPYAZSZOWNLGK2RJQTSBJQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.548841144 (603h3m40.432254761s) [noise=true , address=45.91.116.4:4937 , piece_id=WK5ORWNSEHZMB5A3IR6A2TOIVDY7TG2FDKZIGVOK36SLDT5R7MKQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.557391011 (603h3m40.440804638s) [noise=true , address=104.36.82.8:4972 , piece_id=LDGQX6RQHVIG57FPC3T5P6QRSAZMW6XXMEG5OBP3CNHGPBKR2ORA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:14:40.569693712 (603h3m40.453107329s) [noise=true , address=104.36.82.6:4875 , piece_id=GVZXOQX4CW6SC3OKEPLCPMY6INQBKL2E2UPB4NRDITPYYG3T6WCQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: <nil> Jul 18 12:15:09.384201015 (603h4m9.267614643s) [noise=true , address=77.90.31.10:4517 , piece_id=BF4AEPQD5AJF4V2VLF3OKW3Y4KBBLACEF64R3CPM63AF4AYQGEFA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: context deadline exceeded; context deadline exceeded Jul 18 12:15:09.384430897 (603h4m9.267844524s) [noise=true , address=77.90.31.19:4843 , piece_id=CKH5OODEYI2J5NBAS5CTSWKMV7YX3Y5Q3ACZOJ5XP527FOLIPNWQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 139.59.144.22:53078->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384565274 (603h4m9.267978904s) [noise=true , address=77.90.31.13:4627 , piece_id=4RH6UEARJCYQ2T6HGGMYZ76FMQWMUMPRPRMLCLCOIPAU67CVPKBA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 139.59.144.22:35388->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384616598 (603h4m9.26803027s) [noise=true , address=77.90.31.15:4698 , piece_id=QSEXFMDYV5IYP6YNFWIU4DAIPJ7EVSPENB5ILXPK5ZZ4OJO4MXCA, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384633074 (603h4m9.268046702s) [noise=true , address=77.90.31.12:4590 , piece_id=ZJDAZYZ4FJRUUYJPIYWQVV6BSH6QMHGVX6DG3A7RNI2IRKVTASMQ, seg_pos={0 0}, upload=482161]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:33514->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 18 12:15:09.384666892 (603h4m9.268080522s) [seg_pos={0 0}, upload=482161]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 18 12:15:09.504847872 (603h4m9.388261498s) [upload=482161]: Done uploading segments... Jul 18 12:15:09.504890933 (603h4m9.388304554s) [upload=482161]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.414164459s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)