Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X..X..X........................................] duration: 1.686531575s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:36:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.686560802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X..X..X.......................................] duration: 1.63452514s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:35:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.634555043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 2.317941444s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.317958985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X..X..X......................................] duration: 1.7882769s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:35:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.788302255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 2.322759819s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:35:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.322797941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X..X..X.....................................] duration: 1.677203263s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:35:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.677224639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 2.462969518s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:35:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.46300637s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X..X..X....................................] duration: 1.653967688s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653999506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X..X..X...................................] duration: 1.606546198s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:34:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.606567302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 2.259334258s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:34:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.259359623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X..X..X..................................] duration: 1.545561719s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:33:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.545578394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 2.268307863s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:33:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.268340522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X..X..X.................................] duration: 1.572535945s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:33:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572581272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 2.143582572s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:32:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.143615887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X..X..X................................] duration: 1.714077075s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:32:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.714111969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 2.292608548s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:32:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.29263508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 2.268975034s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:31:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.269084984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X..X..X...............................] duration: 1.574697407s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:31:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.574727405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X..X..X..............................] duration: 1.662511437s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:30:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.662597982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 2.222582266s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.222607729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X..X..X.............................] duration: 1.649523251s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:29:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649546023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 2.366931763s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:29:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.366957189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 2.25074391s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.250786025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X..X..X............................] duration: 1.740363534s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:29:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.740391603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 2.274684967s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.274725089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X..X..X...........................] duration: 1.587306477s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:28:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.587341787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 2.215768547s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:27:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.215791433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X..X..X..........................] duration: 1.652907659s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.652930661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X..X..X.........................] duration: 1.672639598s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:27:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.672693837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 2.293984129s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:26:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.294023123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X..X..X........................] duration: 1.590274699s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:25:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.590296253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 2.20431811s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:25:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.204342706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 2.214714274s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:25:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.214750181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 2.367604957s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:24:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.367634426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X..X..X.......................] duration: 1.72934638s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:24:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.729371364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 2.31866162s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:24:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.318757815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 2.300705198s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:23:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.300746582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X..X..X......................] duration: 1.998566056s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:23:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.998641398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 2.20363473s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.203655919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................X..X..X.....................] duration: 1.572731025s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:23:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572759427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 2.18031236s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:22:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.180364603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................X..X..X....................] duration: 1.647616694s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:22:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.64764415s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................X..X..X...................] duration: 1.663131756s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:22:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663156023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 2.247233488s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:21:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.247262973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 2.34330413s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:21:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.343376561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................X..X..X..................] duration: 1.549072622s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:21:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.549098018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 2.281190413s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:20:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.281280483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................X..X..X.................] duration: 1.639636719s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:20:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.639698965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 2.37652095s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:20:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.376546619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................X..X..X................] duration: 1.621776123s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.621797976s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)