Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X..........XX...........] duration: 2.396167381s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:56:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.396197266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X..........XX..........] duration: 2.40893193s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:56:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.409043624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X..........XX.........] duration: 2.411288962s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:56:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.411384803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.296881154s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:55:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.296913902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X..........XX........] duration: 2.697020652s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:55:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.697103307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.261554833s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:54:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.261651603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.294518026s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:54:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.29454819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X..........XX.......] duration: 2.358587929s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:54:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.358622552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.335788917s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:53:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.335825666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X..........XX......] duration: 2.3055235s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:53:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.305556506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.324321889s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.324353026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X..........XX.....] duration: 2.365234858s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.365264786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.240507766s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:52:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.24053595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X..........XX....] duration: 2.309138188s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:52:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.309168708s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.328344345s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:51:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.328381763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X..........XX...] duration: 2.403399592s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.403429506s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.290718654s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:50:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.290744593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X..........XX..] duration: 2.336747191s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:50:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 2.336774696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X..........XX.] duration: 28.49134271s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 28.491364237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.164359741s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:50:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.164385276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X..........XX] duration: 30.108849324s trace: https://jaeger.dev.storj.io/trace/1d7d54eeb498a75d FAIL (2025-11-10 02:49:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== ObjectBegin Nov 10 02:49:19.532116278 (255h8m22.233708733s) [upload=224386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:49:19.644128091 (255h8m22.345720545s) [seg_pos={0 0}, upload=224386]: Begin upload segment... Nov 10 02:49:19.644340003 (255h8m22.345932493s) [seg_pos={0 0}, upload=224386]: Done begin upload segment. Nov 10 02:49:19.644402179 (255h8m22.345994639s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645499720 (255h8m22.347092232s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.645613893 (255h8m22.347206351s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646048836 (255h8m22.347641297s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646248910 (255h8m22.34784137s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646551101 (255h8m22.348143563s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.646727358 (255h8m22.348319817s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647007264 (255h8m22.348599729s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647171187 (255h8m22.348763647s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647480962 (255h8m22.349073422s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647628502 (255h8m22.349220965s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.647977046 (255h8m22.349569505s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648087175 (255h8m22.349679631s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.648652096 (255h8m22.350244554s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649219321 (255h8m22.350811781s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:19.649785798 (255h8m22.35137826s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Uploading piece... Nov 10 02:49:20.259288970 (255h8m22.960881431s) [upload=224386]: Next returned nil segment Nov 10 02:49:20.259306668 (255h8m22.960899127s) [upload=224386]: Waiting for error group managing segments... Nov 10 02:49:20.395135207 (255h8m23.096727687s) [noise=true , address=77.90.31.12:4592 , piece_id=XWSAHQWQN67QM65I2VVVCRZJSTDVKFA3G5HGTICKHVF42HOHYMIQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.403628447 (255h8m23.105220908s) [noise=true , address=45.91.116.8:4048 , piece_id=AXDYRQY2CFOSRSPKO5L3WV6OLXIRH3FJPU32H7RGKB5HK66ANKAQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408172259 (255h8m23.109764721s) [noise=true , address=77.90.31.10:4517 , piece_id=RB5UZWFLJTX32HVY2NUJTITXVEWUIY5SPQOXDCZR2MEZUECULJXA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.408574031 (255h8m23.110166519s) [noise=true , address=77.90.31.11:4555 , piece_id=HVJM6KKIDIH374OVDJKIK5TCYP4NOJF4URPI4EBMWU2N6CPM455A, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.424130707 (255h8m23.125723168s) [noise=true , address=45.91.116.12:4227 , piece_id=KSFJCS3PIGM75BRFSSYVGIFXIPKOKJNRC5ECZRGY3YL7SNC7EBZQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:20.425417646 (255h8m23.127010107s) [noise=true , address=45.91.116.10:4081 , piece_id=LTS6P6RSN4E566SI4VRD2CAIPR5GOZ2IRC3LOA6STCM2TPZGJDTA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.038913552 (255h8m23.740506008s) [noise=true , address=145.79.227.22:4509 , piece_id=O4UETIMYUQIIFSV7FFK5V7CRAFZHX7RSJCA4XFPBPSUMKTCN72AQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:21.076759645 (255h8m23.778352105s) [noise=true , address=145.79.227.9:4367 , piece_id=FI2AVNSL2LDRSJPFEGMKBIARHPTXHR3V2AJ2XHMTJAYUG4GBJ6NA, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.287420158 (255h8m24.989012617s) [noise=true , address=145.79.227.30:4762 , piece_id=LWKD4I74JLMJGIESNAVR45Z47GYWY4C3PONE457BFEXQNZZERDKQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:22.727800137 (255h8m25.429392596s) [noise=true , address=145.79.227.8:4331 , piece_id=5N6MJVIHVMUYERQ6SPKP3ZYPXNYQ6KZGMGA7BHEOZQZXGMN7NEHQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: <nil> Nov 10 02:49:49.250415030 (255h8m51.952007491s) [noise=true , address=145.79.227.24:4583 , piece_id=JOBX4IRDCJNDQSJPII7EWE7XVABWTCJELEADIRWXE2RJKGNSEFSQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, address:145.79.227.24:4583): protocol: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38582->145.79.227.24:4583: use of closed network connection Nov 10 02:49:49.250591869 (255h8m51.952184329s) [noise=true , address=145.79.227.25:4618 , piece_id=6GRU5E4GS5COAJIUA4NGVN25OGEWQWMJUPDISTW7FD6QFHH3KLMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55746->145.79.227.25:4618: use of closed network connection Nov 10 02:49:49.251142421 (255h8m51.952734882s) [noise=true , address=145.79.227.11:4730 , piece_id=RUTBBLB3RW3237L6JKBH7KFZU66OYEXMXIWTY7BLHZVQGMMOWZ5Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38004->145.79.227.11:4730: use of closed network connection Nov 10 02:49:49.251306839 (255h8m51.952899296s) [noise=true , address=145.79.227.25:4619 , piece_id=734AFPTSYL2FUMFLRJRWY7OHV3H5Q4VDFBPNVOIMQNG4UZ7VLAMQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43660->145.79.227.25:4619: use of closed network connection Nov 10 02:49:49.251731578 (255h8m51.953324035s) [noise=true , address=145.79.227.15:4875 , piece_id=GZN734CEOYPSDV2NK6RLSETBW6N42GMK7U27GICFHZ2MTJRN676Q, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43158->145.79.227.15:4875: use of closed network connection Nov 10 02:49:49.252096232 (255h8m51.953688692s) [noise=true , address=145.79.227.20:4439 , piece_id=D72GBXO35IJR5QUHBPRBPGMPQQXEW3NS5BIHRDVGOW2B5TCPMEPQ, seg_pos={0 0}, upload=224386]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: read tcp 139.59.144.22:57572->145.79.227.20:4439: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:49.252130010 (255h8m51.953722464s) [seg_pos={0 0}, upload=224386]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:49.356977918 (255h8m52.058570378s) [upload=224386]: Done uploading segments... Nov 10 02:49:49.357025760 (255h8m52.058618226s) [upload=224386]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.108870766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.765114191s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:49:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.765136048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.114863723s trace: https://jaeger.dev.storj.io/trace/14afd512f043ed93 FAIL (2025-11-10 02:49:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ntBegin Nov 10 02:48:30.663540777 (255h7m33.365133243s) [seg_pos={0 0}, upload=224375]: Begin upload segment... Nov 10 02:48:30.663603461 (255h7m33.365195918s) [seg_pos={0 0}, upload=224375]: Done begin upload segment. Nov 10 02:48:30.663663440 (255h7m33.365255898s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.663672328 (255h7m33.365264787s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.664683265 (255h7m33.366275722s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.665755413 (255h7m33.367347915s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.666422659 (255h7m33.368015119s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667111268 (255h7m33.36870375s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.667731650 (255h7m33.369324107s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.668817057 (255h7m33.370409515s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670302642 (255h7m33.371895101s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.670811826 (255h7m33.372404295s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671227489 (255h7m33.372819939s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.671776679 (255h7m33.373369136s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672380883 (255h7m33.373973334s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.672815379 (255h7m33.374407836s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673249374 (255h7m33.374841825s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.673471942 (255h7m33.3750644s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674054811 (255h7m33.375647269s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:30.674657562 (255h7m33.376250017s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Uploading piece... Nov 10 02:48:31.232109148 (255h7m33.9337016s) [upload=224375]: Next returned nil segment Nov 10 02:48:31.232128796 (255h7m33.933721252s) [upload=224375]: Waiting for error group managing segments... Nov 10 02:48:31.374785118 (255h7m34.076377575s) [noise=true , address=45.91.116.13:4165 , piece_id=JO6AUEQISEUI7R4SIEEOZPQKLPAHKMT57WT35X2EK3IAOCHO6P5A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.377082753 (255h7m34.078675202s) [noise=true , address=77.90.31.14:4676 , piece_id=KUYDXGENHNFSUGJLKQA7SPGBDVDELBDMUNOJ6BLDMGA4ME66IYGA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.380390392 (255h7m34.081982846s) [noise=true , address=45.91.116.6:4006 , piece_id=SUHYTKSELXVRUOVHG7G465QJAO2R76C67L35WNHC2WJKZJ6JPHWA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385354755 (255h7m34.086947212s) [noise=true , address=45.91.116.4:4962 , piece_id=WGMBNOCMHEJHMDT6YRYQNQXI6BMC36HAWNJINH5UFHE6I2ZENDVA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.385390862 (255h7m34.086983312s) [noise=true , address=45.91.116.10:4084 , piece_id=ERJOCAD7FPVRZYX4YMJJOEILTBGM7FG32BCIVI2J4QNLZ6VMN6WA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.386876749 (255h7m34.088469199s) [noise=true , address=77.90.31.18:4840 , piece_id=4W54TEFSVY5XJOOXH7XACUEDAV2CDA3M3XSWHE4XDDEOPSFYZQTQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.391454363 (255h7m34.093046817s) [noise=true , address=77.90.31.21:4889 , piece_id=LW4GH2LLEFC2QW6WZ3DUSMEREX7ZX4TI2B27TPGD5MAT23FDNPLQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.418981065 (255h7m34.120573521s) [noise=true , address=77.90.31.15:4703 , piece_id=QL7O2MEZ3SNTSV7BAYI67J7UCZRIS54DVFECMVFF3LGQIK2UFKDQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.622445963 (255h7m34.324038421s) [noise=true , address=5.231.73.2:4030 , piece_id=Z67PNEMRP6GGBXCF4M6IANMAVKICGH2TAIM5DIFC42DW2ZL62DQA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.897130275 (255h7m34.598722732s) [noise=true , address=64.49.13.155:4915 , piece_id=LVPNM27AR6H3YBG5JLMT7HHPOYW7V5HPY46KQSS6HYXDSMFSR23A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.955175298 (255h7m34.656767751s) [noise=true , address=145.79.227.15:4881 , piece_id=ZT6V4PXZ4MYL2RBTNS6ZMLNCYPSOJATOGPNKQJILTX7DDRZHVDZA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:31.963277388 (255h7m34.664869842s) [noise=true , address=193.149.164.143:4696 , piece_id=GQTOULVKSYP5QNED33XBBWCZ3P3ZCFGSRBREKEGHC4TERWMSYGNQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:48:32.060058582 (255h7m34.761651041s) [noise=true , address=145.79.227.28:4734 , piece_id=3PUSK7MS7YTWHZQYTRUSJ5CTZOACAJIOIUDHQJA6VXNI7SM3BJHQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: <nil> Nov 10 02:49:30.258586678 (255h8m32.960179139s) [noise=true , address=145.79.227.27:4706 , piece_id=IZQCBIHRNIAJ3T5FFQKTXRCXVG3DNFJQXAZBD2FWKHVEL5V4H5QQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12oXn3dUQ92NpzyQL4GWdJdxyC1uptXSWZ9FZfxQE9FmvRn89GV, address:145.79.227.27:4706): protocol: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36020->145.79.227.27:4706: use of closed network connection Nov 10 02:49:30.258693882 (255h8m32.960286345s) [noise=true , address=145.79.227.52:4456 , piece_id=EHNEY2BCAM6R5SIVJ2W6HKMSI5OBGEMVBMW4BVDFYELVJXQGIXOQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1XUZ2ggL9Q4V8uyJdwvwztqCTvj1DFGRWGtYzKJiaRVaByhfTx, address:145.79.227.52:4456): protocol: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50882->145.79.227.52:4456: use of closed network connection Nov 10 02:49:30.258867006 (255h8m32.960459469s) [noise=true , address=145.79.227.5:4225 , piece_id=FWRAJN3JRTXZYLY7ASLKMRU5ZJQBPQRP6OF6JGVWNDF7YMWXCKIQ, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59150->145.79.227.5:4225: use of closed network connection Nov 10 02:49:30.258961701 (255h8m32.960554165s) [noise=true , address=145.79.227.26:4667 , piece_id=JIVJ33ZMAP2GS45TXFSGISZLXA4TAQK2PF2QSVXSL54KVED4MN3A, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:12iLz6HjQQ1HK7nbPen8szRrZywZcbPTs58FmCPDnAyAEg4AvNk, address:145.79.227.26:4667): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.259228733 (255h8m32.960821196s) [noise=true , address=145.79.227.12:4772 , piece_id=AJPA562JHDVPSO2WOMNQADEQU37T4HTZSB3BWX4FGEBT6DNQXJXA, seg_pos={0 0}, upload=224375]: Done uploading piece. err: ecclient: upload failed (node:1BP2yptVJkJzCKenuKzMZ5sBbVbqXcaoVviwKwrnbxodo1pR5w, address:145.79.227.12:4772): protocol: expected piece hash; context deadline exceeded; EOF Nov 10 02:49:30.260082007 (255h8m32.961674467s) [seg_pos={0 0}, upload=224375]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 10 02:49:30.369468224 (255h8m33.071060686s) [upload=224375]: Done uploading segments... Nov 10 02:49:30.369548300 (255h8m33.071140755s) [upload=224375]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.114894885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..........X] duration: 30.112181054s trace: https://jaeger.dev.storj.io/trace/7d182d81ad8a2bbc FAIL (2025-11-10 02:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== m_ObjectBegin Nov 10 02:48:49.270390205 (255h7m51.971982653s) [upload=224379]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:48:49.380896263 (255h7m52.08248872s) [seg_pos={0 0}, upload=224379]: Begin upload segment... Nov 10 02:48:49.380962799 (255h7m52.082555238s) [seg_pos={0 0}, upload=224379]: Done begin upload segment. Nov 10 02:48:49.381004960 (255h7m52.082597419s) [noise=true , address=145.79.227.17:4972 , piece_id=BSQNYG5LAJ4A7ICQFYH3H4I6TV7QZKD3KE5CGPKVRJX7I77TV5EQ, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.381061892 (255h7m52.082654343s) [noise=true , address=77.90.31.14:4663 , piece_id=2KSB2GCWMSOMNX2EG22GSNE42O7QAWRYN6TXYVVYK7UI4YDEX24Q, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.381619346 (255h7m52.083211807s) [noise=true , address=77.90.31.13:4627 , piece_id=3P6PPFJFMVETNHXWREO2QOLWMHYTUKIUNTRLUQWDSNLKGO3DSMCA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.381970847 (255h7m52.083563301s) [noise=true , address=145.79.227.9:4365 , piece_id=2PT326OOOSE4PACW4DJU424OVHZM5MCAC2Q5R7323FAM2DOCAHDA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.382137550 (255h7m52.083729988s) [noise=true , address=77.90.31.15:4697 , piece_id=SNDY6HW45KHPJLFYHQBVVZ32ENTXDMXGBGYNXTB77AN2LZ4POS4A, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.382341090 (255h7m52.083933541s) [noise=true , address=145.79.227.11:4730 , piece_id=H54Q5DLSTJAQH4IGKQEB7TYKLIDQRRYNRXEDJY4245YFU6ZIOSZA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.382628287 (255h7m52.084220745s) [noise=true , address=145.79.227.13:4804 , piece_id=JTGSQT64LYUHK2XR5CAPV6FN5YJ42ZCZHZ6IEWXA5ANIVZY3YBHQ, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.382749574 (255h7m52.084342032s) [noise=true , address=77.90.31.16:4735 , piece_id=JZQLDULJURUHFEIZBPDJ7IOYROGDBFFTVMP4HQZI2TWJTMNU5X6Q, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.383117125 (255h7m52.084709576s) [noise=true , address=145.79.227.30:4763 , piece_id=G4OD3YAJHYMNCWRWHAFQCOKPC4ISNTQOMTLLIK5AQDBB6N4BBW2Q, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.383321433 (255h7m52.084913871s) [noise=true , address=145.79.227.9:4366 , piece_id=62NBCLSHN6XRJ26Y6WLR2DWFIXC7VNR6YQRD5KFKWTFCNHAYZR2A, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.383561253 (255h7m52.085153701s) [noise=true , address=145.79.227.22:4511 , piece_id=MYTR75D72XRQJNKRXM7U5HXDASCKFRRSIFVA3IIS6U3CO6KGMB3A, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.383677985 (255h7m52.085270439s) [noise=true , address=45.91.116.12:4227 , piece_id=2TH2LGK5ABXAX46OEV2B5XB67GUQDVJYXUHGPYWCXAQNNVYONOEQ, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.383983684 (255h7m52.085576138s) [noise=true , address=145.79.227.16:4909 , piece_id=3TI4H3ER2JHAL4T2RMGI4VHF6ZREJFCCTZZQ7CSQVW2UATGPRXRA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.384019115 (255h7m52.085611566s) [noise=true , address=145.79.227.10:4401 , piece_id=TRJQ2E4PSLSFA6RSV3BJRDR3USUFJTF72CYKBZSAT2IVLDKGZAYA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.384560736 (255h7m52.086153197s) [noise=true , address=145.79.227.15:4875 , piece_id=GIARKVUBXSWV4SJ75UDCPOP5GULE2KHCA5N7LHBZ6EHWU55ENSXA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.386804569 (255h7m52.088397009s) [noise=true , address=145.79.227.20:4439 , piece_id=HCAET7PGUPDLUZRTMVVNXKP5DD2AF2RJAPFPNI6BHCKC3RUQOSBA, seg_pos={0 0}, upload=224379]: Uploading piece... Nov 10 02:48:49.993370338 (255h7m52.694962776s) [upload=224379]: Next returned nil segment Nov 10 02:48:49.993385138 (255h7m52.694977586s) [upload=224379]: Waiting for error group managing segments... Nov 10 02:48:50.130326057 (255h7m52.831918518s) [noise=true , address=45.91.116.12:4227 , piece_id=2TH2LGK5ABXAX46OEV2B5XB67GUQDVJYXUHGPYWCXAQNNVYONOEQ, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.130541203 (255h7m52.832133659s) [noise=true , address=77.90.31.16:4735 , piece_id=JZQLDULJURUHFEIZBPDJ7IOYROGDBFFTVMP4HQZI2TWJTMNU5X6Q, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.131964630 (255h7m52.833557084s) [noise=true , address=77.90.31.13:4627 , piece_id=3P6PPFJFMVETNHXWREO2QOLWMHYTUKIUNTRLUQWDSNLKGO3DSMCA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.132299621 (255h7m52.833892084s) [noise=true , address=77.90.31.14:4663 , piece_id=2KSB2GCWMSOMNX2EG22GSNE42O7QAWRYN6TXYVVYK7UI4YDEX24Q, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.133568862 (255h7m52.835161315s) [noise=true , address=77.90.31.15:4697 , piece_id=SNDY6HW45KHPJLFYHQBVVZ32ENTXDMXGBGYNXTB77AN2LZ4POS4A, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.839084329 (255h7m53.540676801s) [noise=true , address=145.79.227.30:4763 , piece_id=G4OD3YAJHYMNCWRWHAFQCOKPC4ISNTQOMTLLIK5AQDBB6N4BBW2Q, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:50.885431637 (255h7m53.587024096s) [noise=true , address=145.79.227.11:4730 , piece_id=H54Q5DLSTJAQH4IGKQEB7TYKLIDQRRYNRXEDJY4245YFU6ZIOSZA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:51.127348471 (255h7m53.828940925s) [noise=true , address=145.79.227.20:4439 , piece_id=HCAET7PGUPDLUZRTMVVNXKP5DD2AF2RJAPFPNI6BHCKC3RUQOSBA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:52.056351674 (255h7m54.75794413s) [noise=true , address=145.79.227.16:4909 , piece_id=3TI4H3ER2JHAL4T2RMGI4VHF6ZREJFCCTZZQ7CSQVW2UATGPRXRA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:48:55.395161624 (255h7m58.096754063s) [noise=true , address=145.79.227.9:4365 , piece_id=2PT326OOOSE4PACW4DJU424OVHZM5MCAC2Q5R7323FAM2DOCAHDA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: <nil> Nov 10 02:49:18.983675603 (255h8m21.685268053s) [noise=true , address=145.79.227.13:4804 , piece_id=JTGSQT64LYUHK2XR5CAPV6FN5YJ42ZCZHZ6IEWXA5ANIVZY3YBHQ, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: read tcp 139.59.144.22:33588->145.79.227.13:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:49:18.983702891 (255h8m21.685295348s) [noise=true , address=145.79.227.15:4875 , piece_id=GIARKVUBXSWV4SJ75UDCPOP5GULE2KHCA5N7LHBZ6EHWU55ENSXA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 139.59.144.22:55016->145.79.227.15:4875: use of closed network connection; write tcp 139.59.144.22:55016->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55016->145.79.227.15:4875: use of closed network connection Nov 10 02:49:18.983754267 (255h8m21.685346718s) [noise=true , address=145.79.227.17:4972 , piece_id=BSQNYG5LAJ4A7ICQFYH3H4I6TV7QZKD3KE5CGPKVRJX7I77TV5EQ, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 139.59.144.22:46552->145.79.227.17:4972: use of closed network connection; write tcp 139.59.144.22:46552->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46552->145.79.227.17:4972: use of closed network connection Nov 10 02:49:18.983834419 (255h8m21.685426869s) [noise=true , address=145.79.227.9:4366 , piece_id=62NBCLSHN6XRJ26Y6WLR2DWFIXC7VNR6YQRD5KFKWTFCNHAYZR2A, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 139.59.144.22:45846->145.79.227.9:4366: use of closed network connection; write tcp 139.59.144.22:45846->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45846->145.79.227.9:4366: use of closed network connection Nov 10 02:49:18.983872152 (255h8m21.685464604s) [noise=true , address=145.79.227.22:4511 , piece_id=MYTR75D72XRQJNKRXM7U5HXDASCKFRRSIFVA3IIS6U3CO6KGMB3A, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: write tcp 139.59.144.22:36760->145.79.227.22:4511: use of closed network connection; write tcp 139.59.144.22:36760->145.79.227.22:4511: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:36760->145.79.227.22:4511: use of closed network connection Nov 10 02:49:18.983906645 (255h8m21.685499094s) [noise=true , address=145.79.227.10:4401 , piece_id=TRJQ2E4PSLSFA6RSV3BJRDR3USUFJTF72CYKBZSAT2IVLDKGZAYA, seg_pos={0 0}, upload=224379]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 139.59.144.22:58802->145.79.227.10:4401: use of closed network connection; write tcp 139.59.144.22:58802->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:58802->145.79.227.10:4401: use of closed network connection Nov 10 02:49:18.983928811 (255h8m21.685521248s) [seg_pos={0 0}, upload=224379]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 10 02:49:19.093660857 (255h8m21.795253308s) [upload=224379]: Done uploading segments... Nov 10 02:49:19.093707636 (255h8m21.795300108s) [upload=224379]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.11220372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.413283566s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:48:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41331723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.4172914s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:47:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.417322538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.329401627s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:45:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.32943248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.340714432s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:44:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340739026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.31305325s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:44:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.313122189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.343879017s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:43:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.343951113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.313735151s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:42:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.313839051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.338799524s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:41:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338829221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.341970884s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.342014605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.477708065s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.477763296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.650667648s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:40:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65070483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.112313207s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112356246s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.603044206s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:39:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.603164084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.587385133s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:38:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.587418519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.689639467s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:38:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.68969581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.689809403s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:37:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.689861618s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.835695477s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.835727561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.609116304s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:36:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.60921966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.651953746s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:35:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65200696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.720213121s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:34:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.720245317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.666211682s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:33:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.666291553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.773432332s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:33:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.773487818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.722650999s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.722675462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.714442343s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.714515111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.564951796s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:31:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.564974758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.610081837s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:30:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.61010872s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)