Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................X..X..X..........] duration: 1.578427161s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:15:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.578451474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................X..X..X.........] duration: 1.596007383s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:14:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.596036456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 2.43540635s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:14:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.435433052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................X..X..X........] duration: 1.714062389s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:14:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.714172631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 2.319606431s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:13:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.319630611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................X..X..X.......] duration: 1.661027371s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.66112179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 2.24140736s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:12:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.241434292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................X..X..X......] duration: 1.670519s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:12:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.67054754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 2.360186704s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.360217369s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................X..X..X.....] duration: 1.576685629s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:11:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.576739742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 2.282016792s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:11:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.282043179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 2.21691114s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:10:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.21694319s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................................X..X..X....] duration: 1.654190216s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:10:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.654218311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 2.290538079s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:10:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.290566699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................................X..X..X...] duration: 1.598207706s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.598232598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 2.287189785s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.287293854s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................................X..X..X..] duration: 1.59905509s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.599078611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 2.255895036s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:09:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.255929645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................................X..X..X.] duration: 1.657160992s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.657514937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 23.944112911s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.944140444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................................X..X..X] duration: 5.000994505s trace: https://jaeger.dev.storj.io/trace/5e0397fa063b3702 FAIL (2025-12-03 03:07:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001017211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................X..X..] duration: 3.111982495s trace: https://jaeger.dev.storj.io/trace/e98607199dd4b5f FAIL (2025-12-03 03:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.112008043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 2.386178739s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:06:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.386208533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................X..X.] duration: 1.741145316s trace: https://jaeger.dev.storj.io/trace/e98607199dd4b5f FAIL (2025-12-03 03:06:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.741164666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 2.33390762s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:06:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.3339365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 1m0.107429792s trace: https://jaeger.dev.storj.io/trace/3d647576540b8a43 FAIL (2025-12-03 03:06:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9127]: Got next segment. Inline: false Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment... Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment. Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece... Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments... Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil> Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments... Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.107441013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................X..X] duration: 5.000975605s trace: https://jaeger.dev.storj.io/trace/e98607199dd4b5f FAIL (2025-12-03 03:05:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00099806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 1.641424306s trace: https://jaeger.dev.storj.io/trace/65ab1c6a41c1c1a2 FAIL (2025-12-03 03:05:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.641457627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.404031479s trace: https://jaeger.dev.storj.io/trace/56b32a9d61b967ae FAIL (2025-12-03 03:04:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 18m53.839811542s) [upload=459099]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:03:08.610755321 (521h18m53.969413996s) [seg_pos={0 0}, upload=459099]: Begin upload segment... Dec 3 03:03:08.610837824 (521h18m53.969496489s) [seg_pos={0 0}, upload=459099]: Done begin upload segment. Dec 3 03:03:08.610846192 (521h18m53.969504848s) [noise=true , address=45.91.116.7:4038 , piece_id=NR2TSJDDQEGBKXLSV6JW66XUGP3H6F5NURKOWXDFTZTQO42C7F2Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.611001594 (521h18m53.969660268s) [noise=true , address=77.90.31.12:4608 , piece_id=DJNI3WJ5HAXPJX3BCLQAYQEQBXWSMKXTKESNZLSXNHXQBNNMNQFQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.611654245 (521h18m53.97031291s) [noise=true , address=77.90.31.16:4741 , piece_id=VMVYZJYBJ7N6RCLM76IRL46ZL7FOT3GO7GC5A6CUTLKGI7TY5JVA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.612235455 (521h18m53.970894121s) [noise=true , address=77.90.31.19:4876 , piece_id=S4V3OJQ7R42V3ZBLNLT5KU4Z5STYL3KX74PCPDRU3ZM4KEWUAQVQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.612956979 (521h18m53.971615646s) [noise=true , address=45.91.116.11:4132 , piece_id=PJGJKDZLD4JMWUX2XANIJLVHKBCTWV75O7XDK247MVLU32JVY25Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.613622043 (521h18m53.97228071s) [noise=true , address=145.79.227.19:4112 , piece_id=TF3OTWRO5FZ5M2NUPG4NM2NS3RQWSMYKWDEHYZO6FPGQSJRSQMGQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.613834931 (521h18m53.972493599s) [noise=true , address=64.49.13.154:4927 , piece_id=ZL6KXJOEICK7MZUB4V4ALYU4KOCRNV6AECQ7M7ZYLGLSOBGSNDUA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.614369531 (521h18m53.973028194s) [noise=true , address=45.91.116.10:4101 , piece_id=LBWJVBSEHBYH7YZ266HT5ZTER7WYCGX6KDESFPMTFV62WT4GR6EQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.615300148 (521h18m53.97395881s) [noise=true , address=45.91.116.13:4181 , piece_id=74LLKI6ZUHMLSBP6SQVINSQX5CVEED7JVBKLOTGMDDOGMCB3N37Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.615993594 (521h18m53.974652259s) [noise=true , address=77.90.31.12:4622 , piece_id=7DRZ47TZADWILU3KBKPAASYQX4XIENMQT24JXBGNFEBIREWSKNIA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.617176974 (521h18m53.975835636s) [noise=true , address=45.91.116.8:4077 , piece_id=UF23SJGIVK3L5FV6RIRDO5ZDSTXLTMMNMR7WLXFFF272QXIE2FPA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.618447184 (521h18m53.977105848s) [noise=true , address=45.91.116.6:4996 , piece_id=XKKOT4YGL3XNL7PLCE2NRFXNI2N7YXHP6LQC72FVTZS3ETZSIDDQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.619471055 (521h18m53.978129714s) [noise=true , address=145.79.227.52:4448 , piece_id=OTITXGV4H3VAS3B2JBJ55VS3DZQLSTKIBT33MKN4BQHC3YY4ZTTA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.620128570 (521h18m53.978787229s) [noise=true , address=77.90.31.15:4700 , piece_id=72J47ZH3BKJ5LKMDERHNOPWHDLHL7EY7B5TZ2JSWTS6PID5L2MPQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.621374535 (521h18m53.980033195s) [noise=true , address=45.91.116.5:4306 , piece_id=FW5JFZ7HI5PXXKZZ3RWUGIZD4MGEQE725BJ5EKGJES5XFOF4GFAA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.621939838 (521h18m53.980598497s) [noise=true , address=45.91.116.6:4981 , piece_id=ZM6HSJTVOILVZEZUB5MHO2W7G5PX7HS2L2FBYY3IPEMKVWXJHHEA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.622485041 (521h18m53.981143703s) [noise=true , address=64.49.13.169:4707 , piece_id=B6PEZA7ML7J4KHZCJXV7CLDORAPQXYAZIJB4IFOM56GMO2FSU7FA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.623723562 (521h18m53.982382237s) [noise=true , address=145.79.227.13:4832 , piece_id=BLVRPGWT6LSDNTONWCFM3OBDKMJUKVLLTZY2P53M5S2M5JOOHSZQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:09.176593796 (521h18m54.535252459s) [upload=459099]: Next returned nil segment Dec 3 03:03:09.176611644 (521h18m54.535270306s) [upload=459099]: Waiting for error group managing segments... Dec 3 03:03:09.330004667 (521h18m54.688663321s) [noise=true , address=77.90.31.16:4741 , piece_id=VMVYZJYBJ7N6RCLM76IRL46ZL7FOT3GO7GC5A6CUTLKGI7TY5JVA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.332268474 (521h18m54.690927141s) [noise=true , address=145.79.227.13:4832 , piece_id=BLVRPGWT6LSDNTONWCFM3OBDKMJUKVLLTZY2P53M5S2M5JOOHSZQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.337504242 (521h18m54.696162895s) [noise=true , address=77.90.31.12:4622 , piece_id=7DRZ47TZADWILU3KBKPAASYQX4XIENMQT24JXBGNFEBIREWSKNIA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.342048516 (521h18m54.70070717s) [noise=true , address=145.79.227.19:4112 , piece_id=TF3OTWRO5FZ5M2NUPG4NM2NS3RQWSMYKWDEHYZO6FPGQSJRSQMGQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.362465727 (521h18m54.721124381s) [noise=true , address=77.90.31.19:4876 , piece_id=S4V3OJQ7R42V3ZBLNLT5KU4Z5STYL3KX74PCPDRU3ZM4KEWUAQVQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.370252799 (521h18m54.728911457s) [noise=true , address=77.90.31.12:4608 , piece_id=DJNI3WJ5HAXPJX3BCLQAYQEQBXWSMKXTKESNZLSXNHXQBNNMNQFQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.434683746 (521h18m54.793342399s) [noise=true , address=145.79.227.52:4448 , piece_id=OTITXGV4H3VAS3B2JBJ55VS3DZQLSTKIBT33MKN4BQHC3YY4ZTTA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.581399641 (521h18m54.940058298s) [noise=true , address=77.90.31.15:4700 , piece_id=72J47ZH3BKJ5LKMDERHNOPWHDLHL7EY7B5TZ2JSWTS6PID5L2MPQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.867991854 (521h18m55.226650508s) [noise=true , address=64.49.13.154:4927 , piece_id=ZL6KXJOEICK7MZUB4V4ALYU4KOCRNV6AECQ7M7ZYLGLSOBGSNDUA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:10.711439001 (521h18m56.070097654s) [noise=true , address=64.49.13.169:4707 , piece_id=B6PEZA7ML7J4KHZCJXV7CLDORAPQXYAZIJB4IFOM56GMO2FSU7FA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:00.400406853 (521h19m45.759065516s) [noise=true , address=45.91.116.6:4996 , piece_id=XKKOT4YGL3XNL7PLCE2NRFXNI2N7YXHP6LQC72FVTZS3ETZSIDDQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:05.891146371 (521h19m51.249805039s) [noise=true , address=45.91.116.10:4101 , piece_id=LBWJVBSEHBYH7YZ266HT5ZTER7WYCGX6KDESFPMTFV62WT4GR6EQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:08.212886766 (521h19m53.571545429s) [noise=true , address=45.91.116.11:4132 , piece_id=PJGJKDZLD4JMWUX2XANIJLVHKBCTWV75O7XDK247MVLU32JVY25Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: write tcp 139.59.144.22:57650->45.91.116.11:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213070835 (521h19m53.571729493s) [noise=true , address=45.91.116.8:4077 , piece_id=UF23SJGIVK3L5FV6RIRDO5ZDSTXLTMMNMR7WLXFFF272QXIE2FPA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213203754 (521h19m53.571862412s) [noise=true , address=45.91.116.5:4306 , piece_id=FW5JFZ7HI5PXXKZZ3RWUGIZD4MGEQE725BJ5EKGJES5XFOF4GFAA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213320824 (521h19m53.571979489s) [noise=true , address=45.91.116.7:4038 , piece_id=NR2TSJDDQEGBKXLSV6JW66XUGP3H6F5NURKOWXDFTZTQO42C7F2Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:12pDw6bqNwi4PSPjpnMHzAxZFoiFYTnGRUUb1a9u2D22csh6WmW, address:45.91.116.7:4038): protocol: rpc: dial tcp 45.91.116.7:4038: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213482461 (521h19m53.572141124s) [noise=true , address=45.91.116.6:4981 , piece_id=ZM6HSJTVOILVZEZUB5MHO2W7G5PX7HS2L2FBYY3IPEMKVWXJHHEA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, address:45.91.116.6:4981): protocol: rpc: dial tcp 45.91.116.6:4981: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213492621 (521h19m53.572151284s) [noise=true , address=45.91.116.13:4181 , piece_id=74LLKI6ZUHMLSBP6SQVINSQX5CVEED7JVBKLOTGMDDOGMCB3N37Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: write tcp 139.59.144.22:33232->45.91.116.13:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213523606 (521h19m53.572182264s) [seg_pos={0 0}, upload=459099]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 3 03:04:08.315543026 (521h19m53.674201687s) [upload=459099]: Done uploading segments... Dec 3 03:04:08.315589618 (521h19m53.674248284s) [upload=459099]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 2.404051039s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.10569992s trace: https://jaeger.dev.storj.io/trace/56b32a9d61b967ae FAIL (2025-12-03 03:04:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 18m53.839811542s) [upload=459099]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 03:03:08.610755321 (521h18m53.969413996s) [seg_pos={0 0}, upload=459099]: Begin upload segment... Dec 3 03:03:08.610837824 (521h18m53.969496489s) [seg_pos={0 0}, upload=459099]: Done begin upload segment. Dec 3 03:03:08.610846192 (521h18m53.969504848s) [noise=true , address=45.91.116.7:4038 , piece_id=NR2TSJDDQEGBKXLSV6JW66XUGP3H6F5NURKOWXDFTZTQO42C7F2Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.611001594 (521h18m53.969660268s) [noise=true , address=77.90.31.12:4608 , piece_id=DJNI3WJ5HAXPJX3BCLQAYQEQBXWSMKXTKESNZLSXNHXQBNNMNQFQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.611654245 (521h18m53.97031291s) [noise=true , address=77.90.31.16:4741 , piece_id=VMVYZJYBJ7N6RCLM76IRL46ZL7FOT3GO7GC5A6CUTLKGI7TY5JVA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.612235455 (521h18m53.970894121s) [noise=true , address=77.90.31.19:4876 , piece_id=S4V3OJQ7R42V3ZBLNLT5KU4Z5STYL3KX74PCPDRU3ZM4KEWUAQVQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.612956979 (521h18m53.971615646s) [noise=true , address=45.91.116.11:4132 , piece_id=PJGJKDZLD4JMWUX2XANIJLVHKBCTWV75O7XDK247MVLU32JVY25Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.613622043 (521h18m53.97228071s) [noise=true , address=145.79.227.19:4112 , piece_id=TF3OTWRO5FZ5M2NUPG4NM2NS3RQWSMYKWDEHYZO6FPGQSJRSQMGQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.613834931 (521h18m53.972493599s) [noise=true , address=64.49.13.154:4927 , piece_id=ZL6KXJOEICK7MZUB4V4ALYU4KOCRNV6AECQ7M7ZYLGLSOBGSNDUA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.614369531 (521h18m53.973028194s) [noise=true , address=45.91.116.10:4101 , piece_id=LBWJVBSEHBYH7YZ266HT5ZTER7WYCGX6KDESFPMTFV62WT4GR6EQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.615300148 (521h18m53.97395881s) [noise=true , address=45.91.116.13:4181 , piece_id=74LLKI6ZUHMLSBP6SQVINSQX5CVEED7JVBKLOTGMDDOGMCB3N37Q, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.615993594 (521h18m53.974652259s) [noise=true , address=77.90.31.12:4622 , piece_id=7DRZ47TZADWILU3KBKPAASYQX4XIENMQT24JXBGNFEBIREWSKNIA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.617176974 (521h18m53.975835636s) [noise=true , address=45.91.116.8:4077 , piece_id=UF23SJGIVK3L5FV6RIRDO5ZDSTXLTMMNMR7WLXFFF272QXIE2FPA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.618447184 (521h18m53.977105848s) [noise=true , address=45.91.116.6:4996 , piece_id=XKKOT4YGL3XNL7PLCE2NRFXNI2N7YXHP6LQC72FVTZS3ETZSIDDQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.619471055 (521h18m53.978129714s) [noise=true , address=145.79.227.52:4448 , piece_id=OTITXGV4H3VAS3B2JBJ55VS3DZQLSTKIBT33MKN4BQHC3YY4ZTTA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.620128570 (521h18m53.978787229s) [noise=true , address=77.90.31.15:4700 , piece_id=72J47ZH3BKJ5LKMDERHNOPWHDLHL7EY7B5TZ2JSWTS6PID5L2MPQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.621374535 (521h18m53.980033195s) [noise=true , address=45.91.116.5:4306 , piece_id=FW5JFZ7HI5PXXKZZ3RWUGIZD4MGEQE725BJ5EKGJES5XFOF4GFAA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.621939838 (521h18m53.980598497s) [noise=true , address=45.91.116.6:4981 , piece_id=ZM6HSJTVOILVZEZUB5MHO2W7G5PX7HS2L2FBYY3IPEMKVWXJHHEA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.622485041 (521h18m53.981143703s) [noise=true , address=64.49.13.169:4707 , piece_id=B6PEZA7ML7J4KHZCJXV7CLDORAPQXYAZIJB4IFOM56GMO2FSU7FA, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:08.623723562 (521h18m53.982382237s) [noise=true , address=145.79.227.13:4832 , piece_id=BLVRPGWT6LSDNTONWCFM3OBDKMJUKVLLTZY2P53M5S2M5JOOHSZQ, seg_pos={0 0}, upload=459099]: Uploading piece... Dec 3 03:03:09.176593796 (521h18m54.535252459s) [upload=459099]: Next returned nil segment Dec 3 03:03:09.176611644 (521h18m54.535270306s) [upload=459099]: Waiting for error group managing segments... Dec 3 03:03:09.330004667 (521h18m54.688663321s) [noise=true , address=77.90.31.16:4741 , piece_id=VMVYZJYBJ7N6RCLM76IRL46ZL7FOT3GO7GC5A6CUTLKGI7TY5JVA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.332268474 (521h18m54.690927141s) [noise=true , address=145.79.227.13:4832 , piece_id=BLVRPGWT6LSDNTONWCFM3OBDKMJUKVLLTZY2P53M5S2M5JOOHSZQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.337504242 (521h18m54.696162895s) [noise=true , address=77.90.31.12:4622 , piece_id=7DRZ47TZADWILU3KBKPAASYQX4XIENMQT24JXBGNFEBIREWSKNIA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.342048516 (521h18m54.70070717s) [noise=true , address=145.79.227.19:4112 , piece_id=TF3OTWRO5FZ5M2NUPG4NM2NS3RQWSMYKWDEHYZO6FPGQSJRSQMGQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.362465727 (521h18m54.721124381s) [noise=true , address=77.90.31.19:4876 , piece_id=S4V3OJQ7R42V3ZBLNLT5KU4Z5STYL3KX74PCPDRU3ZM4KEWUAQVQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.370252799 (521h18m54.728911457s) [noise=true , address=77.90.31.12:4608 , piece_id=DJNI3WJ5HAXPJX3BCLQAYQEQBXWSMKXTKESNZLSXNHXQBNNMNQFQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.434683746 (521h18m54.793342399s) [noise=true , address=145.79.227.52:4448 , piece_id=OTITXGV4H3VAS3B2JBJ55VS3DZQLSTKIBT33MKN4BQHC3YY4ZTTA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.581399641 (521h18m54.940058298s) [noise=true , address=77.90.31.15:4700 , piece_id=72J47ZH3BKJ5LKMDERHNOPWHDLHL7EY7B5TZ2JSWTS6PID5L2MPQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:09.867991854 (521h18m55.226650508s) [noise=true , address=64.49.13.154:4927 , piece_id=ZL6KXJOEICK7MZUB4V4ALYU4KOCRNV6AECQ7M7ZYLGLSOBGSNDUA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:03:10.711439001 (521h18m56.070097654s) [noise=true , address=64.49.13.169:4707 , piece_id=B6PEZA7ML7J4KHZCJXV7CLDORAPQXYAZIJB4IFOM56GMO2FSU7FA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:00.400406853 (521h19m45.759065516s) [noise=true , address=45.91.116.6:4996 , piece_id=XKKOT4YGL3XNL7PLCE2NRFXNI2N7YXHP6LQC72FVTZS3ETZSIDDQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:05.891146371 (521h19m51.249805039s) [noise=true , address=45.91.116.10:4101 , piece_id=LBWJVBSEHBYH7YZ266HT5ZTER7WYCGX6KDESFPMTFV62WT4GR6EQ, seg_pos={0 0}, upload=459099]: Done uploading piece. err: <nil> Dec 3 03:04:08.212886766 (521h19m53.571545429s) [noise=true , address=45.91.116.11:4132 , piece_id=PJGJKDZLD4JMWUX2XANIJLVHKBCTWV75O7XDK247MVLU32JVY25Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: write tcp 139.59.144.22:57650->45.91.116.11:4132: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213070835 (521h19m53.571729493s) [noise=true , address=45.91.116.8:4077 , piece_id=UF23SJGIVK3L5FV6RIRDO5ZDSTXLTMMNMR7WLXFFF272QXIE2FPA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213203754 (521h19m53.571862412s) [noise=true , address=45.91.116.5:4306 , piece_id=FW5JFZ7HI5PXXKZZ3RWUGIZD4MGEQE725BJ5EKGJES5XFOF4GFAA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213320824 (521h19m53.571979489s) [noise=true , address=45.91.116.7:4038 , piece_id=NR2TSJDDQEGBKXLSV6JW66XUGP3H6F5NURKOWXDFTZTQO42C7F2Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:12pDw6bqNwi4PSPjpnMHzAxZFoiFYTnGRUUb1a9u2D22csh6WmW, address:45.91.116.7:4038): protocol: rpc: dial tcp 45.91.116.7:4038: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213482461 (521h19m53.572141124s) [noise=true , address=45.91.116.6:4981 , piece_id=ZM6HSJTVOILVZEZUB5MHO2W7G5PX7HS2L2FBYY3IPEMKVWXJHHEA, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, address:45.91.116.6:4981): protocol: rpc: dial tcp 45.91.116.6:4981: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213492621 (521h19m53.572151284s) [noise=true , address=45.91.116.13:4181 , piece_id=74LLKI6ZUHMLSBP6SQVINSQX5CVEED7JVBKLOTGMDDOGMCB3N37Q, seg_pos={0 0}, upload=459099]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: write tcp 139.59.144.22:33232->45.91.116.13:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 3 03:04:08.213523606 (521h19m53.572182264s) [seg_pos={0 0}, upload=459099]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 3 03:04:08.315543026 (521h19m53.674201687s) [upload=459099]: Done uploading segments... Dec 3 03:04:08.315589618 (521h19m53.674248284s) [upload=459099]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.105721873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 2.193040819s trace: https://jaeger.dev.storj.io/trace/65ab1c6a41c1c1a2 FAIL (2025-12-03 03:03:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.193155501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001293677s trace: https://jaeger.dev.storj.io/trace/65ab1c6a41c1c1a2 FAIL (2025-12-03 03:03:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001312662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 2.094653644s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:24:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.09468112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 1.670059168s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.670179015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 1.672548661s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:23:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.672574112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.XXX........................................................] duration: 1.668348123s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:22:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.668376263s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..XXX.......................................................] duration: 1.65804535s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:21:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658172502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...XXX......................................................] duration: 1.666262597s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:21:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.666298887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....XXX.....................................................] duration: 1.566024924s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:20:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.566047196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....XXX....................................................] duration: 1.612337036s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:19:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.612376589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......XXX...................................................] duration: 2.107629677s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:19:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.107667127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......XXX..................................................] duration: 2.130598561s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:18:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.130621097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........XXX.................................................] duration: 1.631465312s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.631526289s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........XXX................................................] duration: 2.105758827s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:17:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.105784876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........XXX...............................................] duration: 2.132138701s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.132209576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........XXX..............................................] duration: 1.692302138s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:16:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.692322357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............XXX.............................................] duration: 2.345306168s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:15:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.345331205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............XXX............................................] duration: 1.843653942s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:14:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.843698123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............XXX...........................................] duration: 1.592662121s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:13:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.592691722s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............XXX..........................................] duration: 1.599375599s trace: https://jaeger.dev.storj.io/trace/74a42a90a4bf9e57 FAIL (2025-12-02 23:13:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.599402123s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)