Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.265244297s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:15:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.265277907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........XX....................................] duration: 2.308744989s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:14:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.308777922s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.536773991s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:14:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.536825969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........XX...................................] duration: 2.334277636s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:14:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.334305141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........XX..................................] duration: 2.329393852s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:13:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.329429423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.339007112s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:13:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.339044698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........XX.................................] duration: 2.308277477s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:12:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.308317458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.471830382s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:12:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.471862858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........XX................................] duration: 2.403200455s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:11:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.403264977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..........XX...............................] duration: 2.553408032s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:11:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.553512021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.312717063s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:11:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.312745232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.270002894s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:10:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.270030385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..........XX..............................] duration: 2.46126651s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:10:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.461297413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..........XX.............................] duration: 2.395068324s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:09:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.395102244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.546056279s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:09:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.546096211s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.447464642s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:09:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.447505179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.432395584s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:09:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.432430594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..........XX............................] duration: 2.363997832s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:08:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.36403288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.240009733s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:08:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.240059007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..........XX...........................] duration: 2.431995062s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:08:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.432026102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.418939787s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:07:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.419051321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..........XX..........................] duration: 2.483162498s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:07:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.483192054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.376045958s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:07:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.376085659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..........XX.........................] duration: 2.280813869s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:06:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.280873155s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.426111483s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:06:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.426159069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.382544396s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:05:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.382578795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..........XX........................] duration: 2.419340389s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:05:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.419378205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.377449546s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:04:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.377477014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X..........XX.......................] duration: 2.46013554s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:04:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.460243093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.43022771s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:03:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.43033441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..........XX......................] duration: 2.34622356s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:03:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.346249013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.529805144s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:03:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.529840016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..........XX.....................] duration: 2.384345647s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:03:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.384433315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.225427583s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:02:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.225461032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..........XX....................] duration: 2.346402926s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.346430678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.353549575s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:02:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.353574168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X..........XX...................] duration: 2.46576195s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:02:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.465795286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.377913309s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:01:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.377934408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..........XX..................] duration: 2.412240499s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:00:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.412275305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.283634419s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:00:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.283738428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..........XX.................] duration: 2.380579273s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 03:00:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.380611147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.593816002s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 03:00:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.593851802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..........XX................] duration: 2.390206098s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:59:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.39023666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.347196217s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:59:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.347227687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..........XX...............] duration: 2.367357728s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:59:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.367394131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..........XX..............] duration: 2.367797502s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:58:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.367916303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..........XX.............] duration: 2.427803555s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.427869002s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.537029716s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.537157405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..........XX............] duration: 2.398166583s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:57:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.398223815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.290009115s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:56:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.290042906s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)